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"Selecting  Research  and  Development  Projects" 
by  I.  Vertinsky,  University  of  British  Columbia 

EXECUTIVE  SUMMARY 


\  This  is  an  interim  report  of  a  two  phase  study. 

— ^The  author 1  n -1  ■nng  rufh  is  to  develop  a  methodology 

to  relate  Research  and  Development  programs  of  government  departments 
to  their  objectives.  The  methodology  here  would  combine  simulation 
and  judgment  techniques  to  derive  performance  indicators  reflecting 
government's  objectives,  forecast  the  total  impact  of  R  &  D 
activities  and  evaluate  them  in  terms  of  these  indicators. 


This  will  be  accomplished  in  two  parts;  the  first  part 
consisted  of  a  cross  impact  simulator  -fKSTM which  deals  with 
the  translation  of  R  &  D  activities  into  derived  costs  and  benefits. 


This  simulator  is  used  to  forecast  the  consequences  of  programs  in 
environments  with  complex  interactions. 

In  the  second  part,  "Evaluation  Procedures" ,  the  author  will 
identify  objectives  and  translate  them  into  an  operational  scheme 
of  evaluation  criteria.  In  other  words,  identify  benefit  and  cost 


components  relating  to  objectives.  ^ _ 

The  first  draft  is  linked  directly  to  this  second  part.  It 
serves  as  an  initial  step  in  the  development  of  this  methodology. 
In  this  draft,  the  author  has  given  a  critique  of  the  existing 
literature  i.  e.  A  Survey  of  Project  Selection  Algorithms. 

As  well  he  has  provided  us  with  a  new  project  selection  algorithm 
which  embodies  objectives  and  preferences. 


* Policy  Simulation  Users  Manual 


Chapter  1 


A  SURVEY  OF  PROJECT  SELECTION  ALGORITHMS 


The  problem  of  selecting  research  and  development  pro¬ 
jects  is  an  important  and  difficult  one.  The  future  competi¬ 
tive  position  of  many  companies  depends  on  the  effectiveness  of 
their  research  and  development  program.  The  decisions  that  are 
made  now  will  determine  whether  the  company  will  remain  in 
business,  and  if  so,  what  business  it  will  be  in.  Research  and 
development  programs  are  equally  important  to  governments  in 
their  attempt  to  control,  direct,  and  stimulate  change.  Since 
many  billions  of  dollars  are  spent  each  year  on  research  and 
development  there  is  justifiable  concern  that  this  money  should 
be  invested  wisely. 

Much  of  the  difficulty  in  selecting  research  and  develop 
ment  projects  arises  from  the  uncertainties  surrounding  the 
selection  decision.  The  outcome  of  each  project  (success  or 
failure)^  may  be  uncertain  due  to  technological  considerations. 


It  is  often  difficult  to  classify  the  outcome  of  a 
project  as  success  or  failure.  Howeve^,  for  planning  purposes  a 
project  may  be  considered  successfuj  if  it  achieved  the  goal  it 
set  out  to,  and  unsuccessful  otherwise. 


and  the  value  of  each  project,  even  if  successful,  is  uncertain 
due  to  unforeseeable  future  needs  and  conditions.  Since  more 
projects  are  available  than  can  be  undertaken,  due  to  budget 
and  other  resource  constraints,  the  organization  must  select 
the  ones  which  they  consider  to  be  of  most  value.  However,  the 
goals  toward  which  the  organization  should  be  striving  are  often 
unclear,  and  the  contribution  of  research  and  development  to  the 
achievement  of  these  goals  even  more  vague.  This  makes  the 
selection  problem  difficult.  "If  basic  data  and  clear-cut 
measures  of  the  effectiveness  of  applied  research  were  available 
there  would  appear  to  be  little  difficulty  establishing  an 
objective  basis  for  selecting  projects"  [64]. 

Any  attempt  to  establish  a  selection  procedure  is 
based,  implicitly  or  explicitly,  on  a  model  of  the  decision 
process.  Ideally  this  model  would  include  all  the  significant 
features  of  the  problem.-  However,  since  any  model  is  only  an 
abstraction  of  the  real-world  situation,  it  necessarily  incor¬ 
porates  many  assumptions  and  simplifications.  This  is  especially 
true  when  dealing  with  a  large  number  of  uncertainties.  However, 
despite  the  approximations  inherent  in  the  model  and  the  uncer¬ 
tainty  of  the  data,  the  selection  procedure  may  still  be  valuable 
for  ".  .  .  decisions  have  to  be  made,  usually  on  inadequate  data, 

^Most  project  selection  algorithms  assume  there  is  a 
large  set  of  available  projects  from  which  to  choose.  Whitman 
and  Landau  C82j  and  Gee  C3*0  dispute  this  assumption,  especially 
in  the  chemical  industry,  but  ir  general  it  seems  to  be  a  rea¬ 
sonable  one. 


and  they  will  be  made  intuitively  on  the  data,  whatever  its 
quality.  Anything  that  can  be  done  to  quantify  the  bases  for 
these  decisions  and  to  demonstrate  the  logical  consequences  of 
the  assumptions  is  a  step  in  the  right  direction"  [7], 

Much  of  the  value  resulting  from  the  development  of  a 
selection  procedure  would  come  from  the  model -devel opment  phase 
itself.  The  development  of  a  model  would  force  the  decision 
maker  to  specify,  in  detail,  his  goals  and  any  assumptions  or 
implicit  constraints  he  imposes.  Any  irrationalities  would  then 
be  clearly  illustrated.  An  analysis  of  current  practices  would 
help  to  eliminate  inconsistencies  between  the  decision  maker's 
personal  objectives  and  the  overall  organizational  objectives. 
Current  "rules  of  thumb"  could  be  tested  for  their  rationality, 
and  effectiveness.  Long  range  planning  and  discussion  of  goals 
would  be  stimulated.  Thus,  an  analysis  of  the  decision  problem 
would  result  in  a  better  understanding  of  the  many  uncertainties 
involved  and  their  effect  on  the  results  of  the  decision  process. 
The  selection  procedure  that  would  result  from  the  development 
of  the  model  would  hopefully  facilitate  a  more  consistent 
treatment  of  decision  problems. 

Many  models  and  selection  procedures  have  been  sug¬ 
gested  in  the  literature.  They  may  be  broadly  divided  into  qual¬ 
itative  and  quantitative  approaches.  The  qualitative  methods 
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generally  consist  of  a  check  list  of  “desirable  properties. 
Projects  are  rated  with  respect  to  each  checklist  criterion  as 


simply  "favorable"  or  "unfavorable"  or  on  a  numeric  scale.  To 
be  selected,  a  project's  ratings  must  follow  an  "acceptable" 
pattern,  or  must  meet  certain  specified  minimum  levels  with 
respect  to  each  criterion.  Such  methods  are  imprecise  for  they 
depend  heavily  upon  the  ability  of  the  decision  maker  to  rate 
objectively  and  consistently.  Furthermore,  they  provide  no 
indication  of  the  relative  value  of  the  acceptable  projects  and 
hence  furnish  little  information  or  guidance  concerning  the 
appropriate  funding  level  for  each  project.  The  selection  pro¬ 
cedure  may  also  be  difficult  to  define  and  apply.  What  con¬ 
stitutes  an  "acceptable"  pattern  of  ratings?  What  should  the 
minimum  acceptable  level  of  a  criterion  be?  Is  it  reasonable 
to  reject  a  project  that  does  not  meet  the  minimum  level  in 
one  area  but  far  exceeds  it  in  another?  Clearly  these  questions 
are  important.  Answering  them  is  the  first  step  in  the 
development  of  a  more  precise,  and  sophisticated  decision  method. 

The  quantitative  models  that  have  been  suggested  in 
the  literature  may  be  classified  into  eight  general  categories: 
scoring  models,  linear,  non-linear,  zero-one,  and  utility  models, 
profitability  indices,  risk  analysis  and  decision  trees. 

1 .1  SCORING  MODELS 

The  scoring  models  are  the  least  sophisticated  of  the 

* 

quantitative  methods.  They  use  th'e  same  type  of  ratings  as  the 
« 

qualitative  methods  to  determine  a  numerical  project  score. 
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The  Mottl ey-Newton  method  [64]  rates  each  project  on  a  three 
point  scale  with  respect  to  five  criteria.  The  ratings  are  then 
multiplied  to  produce  the  project  score.  Garguilo  et  al.  [32] 
and  Hertz  and  Carlson  [42]  suggest  a  method  whereby  each  project 
is  rated  as  "favorable,"  "unfavorable,"  or  "no  opinion"  with 
respect  to  the  criteria  which  are  divided  into  three  classes; 
economic,  technical,  and  commercial  factors.  The  number  of 
each  type  of  response  in  each  class  is  counted  and  a  score  for 
that  class  calculated.  The  scores  for  the  three  classes  are 
multiplied  together  to  form  the  overall  project  score. 

The  scoring  models  have  many  of  the  defects  of  the 
qualitative  methods.  They  depend  heavily  on  the  decision  maker's 
rating  ability,  and  provide  only  an  ordinal  ranking  of  projects. 
Since  the  ratings  are  on  an  arbitrary  scale,  it  is  not  possible 
to  know  how  much  better  one  project  is  than  another.  It  may 
not  even  be  possible  to  know  if  the  best  project  is  "good." 

{They  may  all  be  of  little  value  to  the  organization.)  Scoring 
models  do  however  have  several  desirable  features.  Since  much 
reliance  must  be  placed  on  the  decision  maker,  very  little  data 
on  the  project  is  needed.  Thus,  the  method  is  most  useful 
for  decisions  concerning  pure  research  projects  and  projects 
in  their  early  stages  of  development  where  little  concrete 
Information  is  available  on  their  costs  and  benefits.  The 
project  scores  can  also  be  used  to,;he1p  diagnose  a  project's 
weak  points.  They  can  illustrate  those  areas  where  the  project 
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could  be  improved.  Furthermore,  scoring  models  can  be  constructed 
to  include  non-economic  criteria  that  are  difficult  to  quantify 
for  use  in  more  sophisticated  models. 


1.2  LINEAR  MODELS 

Linear  models  often  use  a  type  of  scoring  system  as 
well.  The  methods  suggested  by  Pound  [68]  and  Dean  and  Nishry 
[21]  use  a  weighted  sum  of  project  ratings  as  the  objective 
function.  Pound  determines  the  appropriate  weights  by  interview¬ 
ing  the  decision  maker.  Dean  and  Nishry  suggest  obtaining  them 
from  statistical  analyses  of  past  decisions.  Both  methods 
provide  only  an  ordinal  ranking  of  projects.  Nutt  [66]  has 
developed  a  linear  model  for  selecting  military  projects.  The 
"effectiveness"  of  each  different  project  at  six  discrete  fund¬ 
ing  levels  is  calculated  by  considering  various  military  needs 
and  goals.  A  linear  program  is  solved  and  the  results  indicate 
the  level  at  which  each  project  should  be  funded,  and  the  man¬ 
power  that  will  be  required.  Asher  [1]  suggests  maximizing 
expected  profit  in  an  L.P.  model  which  allocates  a  non-homogeneous 
work  force  to  projects. 

The  linear  models  generally  provide  more  information 

about  funding  levels  and  the  relative  values  of  the  various 

projects.  However  they  require  mof>e  data  to  do  this;  estimates 

* 

of  profit,  or  effectiveness,  and  probabilities  of  success  are 
requi red. 


1 .3  NON-LINEAR  MODELS 


Many  of  the  non-linear  models  view  the  selection  pro¬ 
cedure  as  a  sequential  problem.  At  the  beginning  of  each  planning 
period  old  projects  are  reviewed,  and  new  projects  are  evaluated. 
The  research  and  development  program  for  the  next  period  is 
selected  from  this  collection  of  old. and  new  projects.  This 
view  of  the  problem  leads  naturally  to  a  dynamic  programming  formu¬ 
lation.  Hess  [44]  suggests  a  method  for  maximizing  the  expected 
discounted  net  profit.  A  discount  factor  must  be  specified  and 
estimates  are  required  of  the  total  expected  discounted  gross 
profit  accruing  from  each  project  if  it  is  successful  in  the 

i.  i- 

nl  period,  as  well  as  the  probability  it  will  be  successful  in 
the  nth  period,  for  all  periods  in  the  planning  horizon.  The - 
probability  of  success  is  assumed  to  be  an  exponential  function 
of  the  current  funding  level  and  past  funding  levels  or  current 
funding  level  alone.  The  result  of  the  procedure  is  an  optimal 
funding  level  for  each  project  in  each  planning  period. 

Bobis  et  al.  [7,  9]  have  suggested  modifications  to 
Hess's  method.  They  developed  a  distribution  of  the  cost  of  com¬ 
pleting  a  project  (success  or  failure)  by  requiring  estimates 
of  the  least,  most  likely,  and  greatest  expected  completion  cost. 
The  probability  of  success  in  any  year  is  then  the  probability 

of  completion  at  the  current  expenditure  level  multiplied 

* 

by  the  probability  of  technical,  legal,  engineering  and 
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commercial  success.  The  optimal  allocation  of  funds  is  determined 
from  estimates  of  sales,  costs,  prices,  time  required  for  commer¬ 
cialization  and  probability  of  success.  Since  the  method  is 
so  dependent  on  the  data,  they  suggest  replacing  point  estimates 
by  distributions  and  simulating  to  obtain  a  more  accurate  value. 

Souder  [73]  and  Rosen  [69]  have  attempted  to  simplify 
Hess's  method  by  allowing  each  project  to  be  funded  at  certain 
discrete  values  only,  and  by  assuming  that  the  probability  of 
success  is  a  function  of  the  current  funding  level  alone.  They 
incorporate  the  additional  constraint  that  there  is  a  "minimum 
and  maximum  amount  that  can  be  spent  on  each  project  over  its 
research  and  development  life"  [69], 

These  methods  are  based  on  a  more  realistic  view  of 
the  decision  process.  However  their  onerous  data  requirements 
make  them  difficult  to  use  except  on  commercial  projects  in  an 
advanced  state  of  development.  The  objective  function  of  these 
models  is  to  maximize  profit.  No  other  possible  goals  are 
considered.  Dean  and  Hauser  [20],  however,  have  suggested 
dynamic  programming  methods  for  use  in  a  military  context  which 
optimize  several  different  criteria. 

1.4  ZERO-ONE  MODELS 

The  zero-one  models  such  as  those  devised  by  Minkes 
and  Samuels  [61],  Freeman  [31],  and.Dean  and  Nishry  [21]  are 
all  very  similar.  They  all  propose*maximizing  an  index  of  value 
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subject  to  budget  and  resource  constraints.  Minkes  and  Samuels 
suggest  the  possibility  of  maximizing  expected  present  value 
but  impose  an  additional  constraint  that  the  total  risk  involved 
in  the  program  (weighted  sum  of  the  variances  of  project  return) 
is  less  than  a  specified  amount.  Freeman  uses  an  index  of  value 
which  is  not  necessarily  profit  oriented  and  which  must  be 
developed  by  the  organization  concerned.  He  provides  for  three 
discrete  levels  of  funding  to  be  considered.  Dean  and  Nishry 
develop  two  models;  one  which  maximizes  the  present  value  of 
future  profits  and  another  using  a  scoring-type  approach  which 
maximizes  some  non-economic  measure  of  value. 

The  zero-one  models  are  very  similar  in  approach  to 
the  linear  ones,  and  require  about  the  same  amount  of  data. 

The  ones  which  restrict  the  possible  funding  level  to  one  value 
are  more  appropriate  for  projects  with  a  fairly  well  determined 
cost. 

1 .5  UTILITY  MODELS 

The  utility  models  suggested  by  Cramer  and  Smith  [16] 
and  Green  [40]  are  an  attempt  to  explicitly  handle  risk  considera¬ 
tions.  They  take  into  account  the  fact  that  it  is  more  important 
to  minimize  loss  than  to  maximize  gain.  Cramer  and  Smith  attempt 
to  reduce  the  va 1 ue  of  a  project  to  its »certainty  equivalent  by 
estimating  a  coefficient  of  risk  aversion  and  a  coefficient  of 
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diversification  from  the  utility  function  obtained  from  the 
decision  maker.  Projects  are  then  selected  or  rejected  on  the 
basis  of  their  certainty  equivalent.  Green  also  obtains  a 
utility  function  from  the  decision  maker  and  uses  it  to  aid  in 
the  decision  process. 

1 .6  PROFITABILITY  INDEX  MODELS 

The  profitability  index  models,  Hirsch  and  Fisher  [45], 
Olsen  [67],  Bobis  and  Atkinson  [8],  and  Disman  [23]  are  all 
based  on  the  same  idea;  the  ratio  of  some  measure  of  the  value 
of  the  project  to  some  measure  of  the  cost  is  used  to  indicate 
the  project's  desirability.  The  differences  lie  in  the  measures 
of  value  and  cost  used.  Disman's  method  is  considered  useful 
[23].  He  suggests  calculating  the  maximum  expenditure  justified 
(MEJ)  which  is  the  present  value  at  some  acceptable  rate  of 
return  of  the  income  generated  by  the  project.  This  is  mul¬ 
tiplied  by  the  probability  of  technical  and  commercial  success 
and  divided  by  the  total  estimated  research  and  development 
costs.  Several  other  more  specialized  indices  have  been  devised 
[36]. 

Profitability  index  methods  are  similar  to  scoring 
models  in  that  they  provide  a  single  numeric  measure  of  the 
desirability  of  each  project,  and  an  cardinal  ranking  of  projects. 
Whereas  scoring  methods'  are  most  suitable  for  pure  research 
projects  and  those  projects  in  their  early  stages  of  development 
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which  have  only  imprecise  data  available,  profitability  index 
models  are  most  suitable  for  commercial  projects,  and  projects 
near  completion,  where  accurate  estimates  can  be  made  of  costs 
and  benefits. 

1.7  RISK  ANALYSIS 

Risk  analysis  is  a  simulation  technique  for  determin¬ 
ing  the  probability  distribution  of  return  on  a  project  [41,  59, 
80].  Distribution  functions  must  be  specified  for  each  factor 
which  affects  return.  Malloy  [59]  suggests  using  a  beta  dis¬ 
tribution  so  that  only  estimates  of  the  lowest,  most  likely, 
and  highest  possible  values  are  required  to  define  the  distri¬ 
butions.  The  project's  development  is  simulated  by  choosing 
a  value  for  each  factor  according  to  its  distribution  and  com¬ 
bining  these  values  in  the  appropriate  way.  Bobis  et  al. 

[7,  9]  have  suggested  using  a  , similar  technique  to  determine 
more  reliable  estimates  of  the  factors  required  in  their  non- 
1  inear  model  . 

1 .8  DECISION  TREES 

Many  of  the  decision  tree  models  use  risk  analysis  as 
a  solution  technique.  A  decision  tree  is  a  graphical  represen¬ 
tation  of  the  expected  stages  of  project  development.  Each 
future  decision  point  and  chance  outcome  point  is  represented  by 
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a  node  in  the  tree.  Hespos  and  Strassman  [43]  suggest  using 
risk  analysis  on  each  possible  path  through  the  tree  (if  there 
are  not  too  many),  or  eliminating  some  paths  by  dominance,  and 
analyzing  the  rest  in  more  detail.  Lockett  and  Freeman-^ 55] 
and  Lockett  and  Gear  [56]  suggest  sampling  at  each  chance  out¬ 
come  point,  and  reducing  the  problem  to  a  deterministic  linear 
program.  This  procedure  is  repeated  many  times,  resulting  in  a 
set  of  feasible  programs  which  are  optimal  for  one  particular 
state  of  the  world.  The  final  program  is  selected  by  examining 
this  set  for  projects  which  are  always  selected  or  never 
selected  and  for  other  significance  patterns.  An  integer  pro¬ 
gramming  method  of  solving  the  decision  tree  problem  directly 
(without  simulation)  is  given  by  Gear  and  Lockett  [33].  It 
becomes  unsolveable  however,  when  there  are  many  chance  outcome 
points . 

1.9  LIMITATIONS  OF  THE  MODELS 

Very  few  of  the  models  and  selection  procedures  that 
have  been  suggested  in  the  literature  have  actually  been  im¬ 
plemented.  Baker  and  Pound  [5]  suggest  two  reasons  for  this: 
lack  of  testing  and  computational  experience,  and  lack  of  realism 
in  many  of  the  models. 

The  most  clearly  unreal  is'bic^  feature  of  many  of  the 
models  is  the  objective  function.  The  goal  of  profit  maximiza¬ 
tion  is  the  only  one  considered  in  most  cases.  In  a  study  of 
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the  utility  and  acceptability  of  project  selection  models, 

Souder  found  that  this  need  not  necessarily  be  the  only,  nor 
even  the  primary  goal.  In  his  study,  "none  of  the  seven  admini¬ 
strators  interviewed  indicated  a  strong  proclivity  to  pursue 
the  maximization  of  expected  values.  Several  administrators 
indicated  a  definite  rejection  of  such  objectives.  All  the 
administrators  viewed  several  non-r.ionetary  goals  as  paramount 
considerations.  Some  administrators  indicated  that  various 
intrinsic  properties  of  the  portfp .  1  .y.  themselves  could  be  more 
important  considerations  than  the  short  term  profitability 
statistics"  [78].  The  scoring-type  models  and  some  of  the  linear 
models  consider  other  types  of  goals.  However  the  method  of 
combining  them  into  an  objective  function  is  generally  quite 
arbitrary  and  thus  their  relative  importance  in  the  model  is 
not  the  same  as  their  relative  importance  in  the  eyes  of  the 
decision  maker.  What  is  needed  then,  is  a  technique  to  combine 
any  of  the  possible  goals  into  an  objective  function  in  accordance 
with  the  decision  maker's  priorities.  "The  assumption  that 
there  exists  an  optimal  solution  or  a  set  of  optimal  solutions 
to  a  problem  involving  multiple  criteria  implies  the  existence 
of  some  preference  ordering  defined  over  the  set  of  feasible 
values  of  the  criteria"  [24].  The  problem  is  then  reduced  to 
one  of  finding  this  "preference  ordering,"  a  subject  which  is 
discussed  in  the  next  chapter.  *  * 


Another  unrealistic  feature  of  many  of  the  models 
concerns  the  assumption  made  about  funding  levels.  The  zero- 
one  models  assume  funding  is  possible  at  only  one  level.  Many 
of  the  linear  and  non-linear  models  assume  funding  is  possible 
at  any  level.  A  more  realistic  approach  would  be  to  allow  a 
project  to  be  funded  in  a  range  or  not  at  all.  Freeman  [31] 
considers  this  idea  but  does  not  implement  it.  He  suggests 
that  there  is  a  "critical  cost  level"  below  which  the  value 
and  probability  of  success  of  a  project  is  very  small,  and  a 
"satiation  point"  above  which  additional  funding  creates  little 
additional  value  and  increases  the  probability  of  success  by  an 
insignificant  amount.  These  upper  and  lower  bounds  could  also 
be  dictated  by  organizational  policy.  There  may  be  an  upper 
limit  on  the  amount  that  may  be  risked  on  any  one  project  and 
a  lower  limit  determined  by  the  least  amount  that  is  "reasonable" 
to  invest  in  a  project. 

The  probability  of  success  of  a  project  is  directly 
related  to  its  funding  level.  Most  of  the  more  sophisticated 
models  require  a  subjective  estimate  of  the  probability  of 
success  of  each  project  at  a  given  level.  This  estimate  is  used 
to  develop  the  relationship  between  funding  and  probability  of 
success.  A  study  by  Souder  [74]  on  the  validity  of  subjective 
probability  of  success  estimates,  suggests  that  they  are  generally 

4 

valid  and  reliable  although  they  may  rtot  always  be  accurately 
communicated  due  to  ulterior  motives  on  the  part  of  the  decision 
maker  and  organizational  pressures. 


The  project  selection  model  which  will  be  developed 
in  the  following  chapters  will  attempt  to  describe  the  selection 
process  more  realistically  by  including  multiple  criteria, 
funding  ranges,  and  probability  of  success  estimates. 


Chapter  2 


RATIONAL  DECISION  MAKING  WITH  MULTIPLE  CRITERIA 


Rational  decision  making  implies  a  goa 1 -d i rected  choice 
among  alternative  courses  of  action.  Goal -di rected  choice 
among  alternatives  requires  both  a  knowledge  of  the  correspondence 
between  actions  and  outcomes,  and  a  subjective  preference  order¬ 
ing  among  outcomes.  This  chapter  focuses  on  the  latter,  the 
development  of  methods  for  evaluating  outcomes  in  terms  of  goals 
and  ordering  them  on  the  basis  of  their  subjective  value. 

Outcomes  can  be  described  by  a  set  of  variables  or 
attributes,  which  reflect  the  dimensions  through  which  the  out¬ 
come  contributes  to  or  detracts  from  the  ultimate  goals  or 
objectives  of  the  decision  maker.  For  example,  in  choosing  a 
house,  the  alternatives  can  be  presented  in  terms  of  such  attri¬ 
butes  as  space,  price,  convenience  of  location,  condition  ana 
facilities,  etc.,  which  describe  each  house  completely  with 
respect  to  the  decision  maker's  goals,  and  thus  define  criteria 
for  evaluation.  These  criteria  may  correspond  directly  to  the 
goals,  or  they  may  be  simply  indicators  which  are  related  to 
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the  goals.  For  another  example  consider  a  government  agency 


that  wishes  to  fund  a  number  of  projects.  Two  of  its  goals 
might  be  to  increase  employment  and  to  maintain  Canadian 
sovereignty.  One  of  the  attributes  used  in  this  case  for  project 
evaluation  corresponds  directly  to  the  first  goal,  i.e.  the 
number  of  jobs  created.  The  second  goal  can  be  measured  in 
several  ways,  for  example,  by  (1)  the  resulting  percentage 
increase  in  Canadian  ownership  of  firms,  and  (2)  the  increase 
in  the  amount  of  local  raw  materials  processed  in  Canada.  These 
last  two  criteria  are  merely  indicators  which  are  related  to 
the  second  goal. 

Multiple  criteria  decision  problems  have  a  well- 
defined  solution  if  there  exists  a  preference  orderino  de¬ 
fined  over  the  feasible  values  of  the  criteria  which  is  complete 
and  transitive.  Completeness  means  that  all  alternatives  can 
be  compared  and  the  one  with  the  greatest  relative  value  can 
be  found.  This  may  be  a  difficult  requirement  in  practice. 

The  decision  maker  may  be  able  to  choose  between  two  alternatives 
which  give  him  $500  or  $800,  but  may  find  it  more  difficult  to 
choose  between  alternatives  which  give  him  $500  or  a  trip  to 
Hawaii.  In  the  first  case  he  need  only  compare  the  levels  of 
the  relevant  attribute  (money).  In  the  second  case  he  must 
relate  the  different  attributes  (money,  trip)  to  an  underlying 
goal  (possibly  prestige)  in  order  to  determine  which  one  has 
the  greater  relative  value.  In  the  context  of  the  project 
selection  problem  the  alternatives  (programs  of  projects)  will 


generally  differ  along  the  level  of  the  relevant  attributes 
rather  than  by  having  different  attributes  altogether.  There¬ 
fore  they  will  tend  to  be  easier  to  compare. 

Transitivity  means  that  if  alternative  B  is  preferred 
to  alternative  A  (the  relative  value  of  B  is  greater  than  the 
relative  value  of  A),  and  alternative  C  is  preferred  to  alterna¬ 
tive  B,  then  alternative  C  is  preferred  to  alternative  A.  This 
is  a  reasonable  assumption  when  all  alternatives  are  readily 
comparabl e  ,  i . e  .  completness  holds. 

In  many  situations  all  that  is  required  of  the  decision 
maker  is  that  he  rank  his  alternatives,  by  considering  each 
criterion  and  the  relationships  between  them.  However,  in  the 
case  of  a  project  selection  decision,  the  number  of  alternatives 
is  large.  An  alternative  in  this  case  is  a  portfolio  of  R&D 
projects  with  a  specified  funding  level  for  each.  Each  project 
under  consideration  is  not  simply  one  possible  component  of  a 
research  and  development  program.  It  is  a  representative  of  a 
set  of  possible  components,  distinguished  by  their  funding 
levels.  Each  funding  level  implies  a  different  probability  of 
success  for  the  project,  and  thus  results  in  a  different  amount 
of  the  attributes  of  that  project.  Therefore  there  is  an 
infinite  number  of  possible  research  and  development  programs, 
and  it  is  impossible  to  solve  the  selection  problem  by  simply 

ranking  the  alternatives.  What  is  required  is  an  explicit  pref- 

« ■ 

erences  ordering  that  may  be  used  to  find  an  optimal  portfolio 
of  projects. 
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There  are  many  methods  of  determining  preference 
orderings  described  in  the  literature.  These  may  be  classified 
into  sequential  elimination  methods,  spatial  proximity  methods, 
mathematical  programming  methods,  and  utility  function  methods 
(directly  assessed  preference  techniques  and  inferred  preference 
techniques).^  Table  1  summarizes  the  classification  and  lists 
the  methods  which  fall  into  each  category. 

2.1  SEQUENTIAL  ELIMINATION  METHODS 

Sequential  elimination  methods  order  the  alternatives 
by  comparing  them  either  to  each  other  or  to  a  standard.  If 
alternatives  are  compared  to  each  other,  those  which  provide 
less  of  all  criteria  than  another  can  be  eliminated.  This 
dominance  technique  is  used  by  Terry  [79]  as  an  initial  filter 
in  selecting  new  product  areas.  Generally  very  few  alternatives 
can  be  eliminated  this  way.  Other  methods  must  be  used  to 
determine  the  ordering  of  the  remaining  alternatives. 

Another  method  of  ordering  alternatives  by  comparing 
them  to  each  other  is  1  ex i cogra phy .  The  alternatives  are  ranked 
on  the  basis  of  their  rating  on  the  most  important  criterion. 

If  any  of  the  alternatives  prove  to  be  equal  in  value,  then  the 
rating  on  the  next  most  important  criterion  is  considered. 


This  classification  is  essentially  the  one  proposed 
by  MacCrimmon  [58]. 
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Elimination  by  aspects  is  similar  to  lexicography  except  that 
instead  of  choosing  the  most  important  criterion  as  the  basis 
for  comparison,  the  one  with  the  most  discriminatory  power 
(greatest  range,  fewest  ties,  etc.)  is  chosen.  These  two  methods 
of  handling  multiple  criteria  seem  to  have  been  developed  because 
they  are  convenient  rather  than  because  they  model  any  conscious 
or  logical  strategy  for  finding  the  best  alternative. 

There  are  two  methods  of  comparing  alternatives  to  a 
standard.  One  is  to  examine  each  alternative  for  its  "worst" 
criterion  level,  and  accept  or  reject  it  on  the  basis  of  whether 
or  not  this  worst  level  meets  the  standard.  This  conjunctive 
strategy  results  in  a  set  of  acceptable  alternatives  which 
exceed  the  standard  with  respect  to  each  criterion.  If  a  govern¬ 
ment  agency  was  concerned  with  keeping  its  fund  granting  program 
free  of  public  criticism,  it  might  adopt  such  a  decision  pro¬ 
cedure  to  prevent  an  obviously  poor  performance  with  respect 
to  any  one  criterion. 

The  other  method  is  a  disjunctive  strategy.  In  this 

case  the  best  criterion  level  of  each  alternative  is  found,  and 

if  it  exceeds  the  standard,  then  the  alternative  is  acceptable. 

This  strategy  results  in  a  set  of  alternatives  which  exceed  the 

standard  with  respect  to  at  least  one  criterion.  A  research 

centre  that  wishes  to  develop  an  excellent  reputation  might  opt 

* 

to  use  such  a  strategy  to  recruit  scientists  on  the  basis  of 
their  strongest  field  of  endeavor. 
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In  each  case  the  problem  of  ordering  the  accepted 
alternatives  remains,  and  must  be  resolved  using  one  of  the 
other  methods.  Because  of  the  restrictive  logic  used  in  these 
methods,  they  are  applicable  in  a  limited  number  of  cases  only. 
The  resulting  acceptable  set  of  alternatives  is  very  different 
with  the  two  methods.  Dawes  [17]  discusses  the  characteristics 
of  groups  of  people  selected  for  various  positions  by  the  two 
methods . 

The  sequential  elimination  methods  are  non-compensa¬ 
tory.  A  high  level  of  achievement  with  respect  to  one  criterion 
cannot  compensate  for  a  deficiency  in  another  area.  In  many 
situations  this  assumption  is  too  restrictive.  However  there 
are  cases  where  these  techniques  have  been  effective.  Kleinmuntz 
[52]  uses  a  sequence  of  elimination  methods  of  this  type  to 
model  a  clinical  psychologist  judging  test  profiles  (MMPI)  as 
being  "normal"  or  "abnormal."  Smith  and  Greenlaw  [72]  use  a 
similar  technique  to  model  a  psychologist  selecting  applicants 
for  jobs.  In  both  studies  the  subjects  were  asked  to  describe 
the  techniques  they  were  using  as  they  made  a  set  of  decisions. 

Their  descriptions  and  decisions  were  examined  to  derive  a  set 
of  sequential  rules  which  could  be  applied  to  other  problems. 


2.2  SPATIAL  PROXIMITY  METHODS 

“ — —  -■■■■-  .  ■  ■  <r 

Spatial  proximity  methods  are  ordering  techniques 
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which  rely  more  heavily  on  a  geometric  representation. 

The  indifference  map  technique  obtains  indifference 
curves  for  each  pair  of  criteria.  An  indifference  curve 
is  a  set  of  points,  each  having  two  co-ordinates,  which 
represent  the  levels  of  the  two  criteria.  All  combin¬ 
ations  of  the  two  criteria  which  are  seen  by  the  decision 
maker  as  having  the  same  relative  value  are  located  on 
the  same  indifference  curve.  MacCrimmon  and  Toda  [57] 
describe  an  efficient  method  of  constructing  indifference 
curves.  One  possible  combination  of  the  two  criteria, 
say  (a,b),  is  chosen  as  a  reference  point.  Since  it 
is  assumed  that  a  higher  level  of  any  criterion  is  always 
preferred  to  a  lower  one,  all  points  "north  east"  of 
the  reference  point  are  more  valuable  than  the  reference 
point,  and  can  be  thought  of  as  the  "accept  region." 

All  points  "south  west"  of.  the  reference  point  are  less 
valuable,  and  can  be  thought  of  as  the  "reject"  region. 
These  areas  may  be  blocked  off  and  excluded  from  further 
consideration  as  in  Figure  2.1.  A  circle  is  drawn 
around  (a,b,).  The  decision  maker  is  then  asked  to 
compare  in  turn,  the  mid-point  of  each  of  the  areas 
which  fall  in  the  open  region  to  the  reference  point. 

If  the  mid-point  is  accepted  (rejected)  more  points  on 

*r 

the  arc  in  the  direction  of  the*reject  (accept)  region 
are  compared  to  the  reference  point  until  one  of  them 
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is  rejected  (accepted).  Each  time  a  point  is  compared  to  the 
reference  point,  another  area  may  be  blocked  off  as  belonging 
to  either  the  accept  or  reject  region.  This  process  is  continued 
until  the  open  area  is  narrow  enough  for  an  indifference  curve 
to  be  drawn  through  it.  All  points  on  the  indifference  curve 
would  have  approximately  the  same  subjective  value  as  the  reference 
point  (a,b).  Other  indifference  curves  for  the  same  two  criteria, 
corresponding  to  different  value  levels  are  derived  by  choosing 
a  different  reference  point.  The  same  technique  is  applied  to 
all  other  pairs  of  criteria  to  derive  a  complete  indifference 
map.  The  alternatives  can  then  be  positioned  in  this  multi¬ 
dimensional  space  with  respect  to  the  indifference  curves  and 
a  complete  ordering  among  them  obtained. 

Mul ti -dimensional  scaling  is  another  proximity  ordering 
technique.  An  ideal  alternative  is  assumed  to  exist.  The  value 
of  any  other  alternative  is  assumed  to  be  inversely  proportional 
to  the  "distance"  of  that  alternative  from  the  ideal  point. 

Klahr  [51]  describes  a  method  of  locating  the  real  and  ideal 
al ternat ives  i n  a  multi-dimensional  space.  He  first  requires  the 
decision  maker  to  judge  the  similarity  of  a  set  of  alternatives. 

The  alternatives  are  then  positioned,  consistent  with  the  near¬ 
ness  assumption.  This  can  always  be  done  in  an  N-l  dimensional 
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space,  where  N  is  the  number  of  alternatives.  The  next  step 
is  to  iteratively  reduce  the  dimensionality  of  the  space  until 
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it  is  less  than  or  equal  to  the  number  of  criteria,  since  one 
would  expect  to  need  no  more  than  one  dimension  for  each  criterion 
that  is  significant  in  the  decision  making  process.  The  dimen¬ 
sion  finally  chosen  is  the  one  that  minimizes  stress  (a  measure 
of  departure  from  perfect  fit).  Klahr  applies  this  technique 
to  the  problem  of  graduate  admission  decisions.  The  method 
however  is  computationally  difficult  and  has  not  been  particu¬ 
larly  successful  . 

2.3  MATHEMATICAL  PROGRAMMING  METHODS 

There  are  two  mathematical  programming  methods:  goal 
programming  and  an  interactive  tradeoff  technique.  Goal  pro¬ 
gramming  requires  the  decision  maker  to  specify  desired  or 
acceptable  levels  of  a  set  of  goals.  A  goal  may  correspond  to 
one  of  the  criteria  or  may  be  combinations  of  the  criteria. 

The  amount  of  dissatisfaction'  accompanying  any  alternative  which 
does  not  satisfy  the  goals  is  determined  by  obtaining  weighting 
coefficients  for  deviations  in  each  direction  from  each  goal. 

The  weighted  sum  of  these  deviations  is  used  as  the  objective 
function  in  a  standard  LP  minimization  procedure.  The  result¬ 
ing  optimal  solution  is  the  criterion  levels  of  the  best  alter¬ 
native.  Lee  and  Clayton  [54]  apply  the  technique  to  the 
scheduling  of  an  academic  department* 
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The  interactive  tradeoff  technique  assumes  that  a 
global  objective  function  exists  but  does  not  require  it  to  be 
defined.  At  any  feasible  alternative  the  decision  maker's 
tradeoffs  among  the  criteria  in  the  neighbourhood  of  the  alter¬ 
native  are  determined.  The  decision  maker  is  asked  to  choose 
between  alternatives  of  the  form  (flt  .  .  .  f  )  and  (fj  +  Aflt 
f 2 *  ■  •  •  »  fj  +  Af ^ ,  .  .  .  fr)  where  f ..  are  the  criterion 
levels.  The  Afi  and  Af^  are  varied  until  the  decision  maker  is 
indifferent  between  the  two  alternatives.  (This  technique  is 
similar  to  the  method  used  in  deriving  indifference  curves.) 

Then  the  tradeoff  between  criterion  i  and  criterion  1, 
w.j  *  -Afi/Af.  is  used  in  approximation  to  the  gradient  of 
the  global  utility  function  at  the  current  point.  This  approxi¬ 
mation  Is  used  in  the  objective  function  of  a  mathematical 
programming  algorithm  which  determines  the  best  direction  to 
move,  i.e.  the  direction  in  which  better  points  than  the  current 
one  lie.  A  set  of  points  at  various  distances  from  the  current 
point  in  that  direction  is  then  generated  and  the  decision  maker 
is  asked  to  choose  the  best  one.  If  this  new  alternative  is 
"good  enough"  the  procedure  terminates,  otherwise  it  is  repeated. 
Geoffrion,  Dyer,  and  Feinberg  [25,  35]  have  developed  this  tech¬ 
nique  and  applied  it  to  the  scheduling  of  an  academic  department. 

These  techniques  construct  the  best  alternative 
rather  than  choosing  it  from  a  set  of  explicitly  pre-defined  alt¬ 
ernatives.  They  can  only  be  used  in  »si  tuations  such  as  a  project 
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selection  problem  where  the  criteria  take  on  continuous  values. 
Applying  these  techniques  to  the  project  selection  problem 
results  in  the  optimal  combination  of  criterion  levels. 


2.4  UTILITY  FUNCTION  METHODS 

The  last  method  of  dealing  with  multiple  criteria  to 
be  considered  here  is  the  use  of  utility  functions.  In  the 
literature  the  term  "utility  function"  has  a  specific  meaning 
which  depends  on  the  axiomatic  system  being  followed.  However, 
here  it  will  simply  mean  a  mapping  which  assigns  to  an  alterna¬ 
tive  a  real  number  (utility)  which  indicates  the  relative  worth 
of  the  alternative.  This  mapping  has  the  two  properties  men¬ 
tioned  previously;  it  is  complete  and  it  is  transitive. 

There  are  two  main  classes  of  methods  used  in  deriving 
utility  functions;  directly  assessed  preferences  and  inferred 
preferences.  Fishburn  [29]  describes  24  methods  of  directly 
assessing  preferences.  The  basic  ideas  behind  these  methods 
are  discussed  below. 

2.4.1  Directly  Assessed  Preference  Techniques 

All  of  the  methods  assume  that  the  utility  of  an  al¬ 
ternative  is  the  sum  of  the  utilities  of  the  individual  criterion 

* 

levels,  i.e.  * 
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U  (alternative)  =  u^x^  +  u2(x2)  +  ...  +un(xn) 
where  the  x..  are  the  levels  of  the  criteria  achieved  by  the 

t*  h 

alternative  and  is  the  utility  function  for  the  i  criterion. 
Fishburn  [28,29,30]  and  v.  Winterfeldt  and  Fischer  [83]  discuss 
the  conditions  under  which  this  additivity  assumption  holds. 

The  directly  assessed  preference  methods  find  utility 
functions  ui  for  each  of  the  criteria.  The  simplest  method 
is  the  direct  rating  method.  The  decision  maker  is  asked  to 
assign  a  utility  to  each  criterion  level  by  rating  it  on  an 
arbitrary  scale.  The  other  methods  ask  the  decision  maker 
questions  on  his  preferences,  then  attempt  to  assign  utilities 
consistent  with  his  responses. 

Ranking  methods  ask  for  an  ordering  of  the  feasible' 
criterion  levels.  This  generally  can  be  reduced  to  a  series  of 
pairwise  comparisons.  For  example,  to  find  the  most  preferred 
criterion  level  of  n  feasible  ones,  the  decision  maker  may 
choose  a  candidate,  then  compare  this  level  with  each  other 
feasible  level  in  turn,  in  order  to  verify  the  hypothesis  that 
it  is  more  preferred.  Utilities  are  then  assigned  to  each 
feasible  criterion  level  in  accordance  with  this  ranking.  One 
way  would  be  to  assign  the  most  preferred  level  a  utility  of 
n,  the  next  preferred  n-1  ,  etc. 

Gamble  methods  require  the  decision  maker  to  choose 

between  gambles  made  up  of  va  r  ious'  cr  i  ter  i  on  levels  which  may 

* 

occur  with  various  probabilities.  One  such  method  uses  the 
least  and  most  desirable  level  of  a  criterion,  say  and 
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respectively.  For  a  set  of  other  criterion  levels  iz^),  the 
decision  maker  is  asked  to  estimate  the  probability  p(z..)  for 
which  zi  has  the  same  value  to  him  as  a  gamble  between  xi  and 
where  yi  occurs  with  probability  p(z^)  and  x.  occurs  with 
probability  1  -  p(z,.).  The  utility  of  criterion  level  z^  is 
then  p  (z-jjufy^)  +  [1  -  p(z..)]  u  (x^).  The  utility  of  levels 
Xj  and  yi  can  be  arbitrarily  set  (for  example  to  1  and  100) 
and  the  utility  of  z^  determined. 

Ordered  metric  methods  require  a  ranking  of  the  dif¬ 
ferences  between  criterion  levels  which  are  adjacent  in  utility. 
An  initial  ranking  of  the  criterion  levels  is  necessary.  One 
example,  the  direct  ordered  metric  method,  requires  the  decision 
maker  to  rank  the  utility  differences  between  four  adjacent 
criterion  levels.  Suppose  u  (a)  <  u(b)  <  u(c)  <  u(d)  and 
0  <  [u(d)  -  u(c)]  <  [u(c)  -  u(b)]  <  [u(b)  -  u ( a )  ] .  Numerical 
assignments  consistent  with  this  metric  ranking  may  be  made. 

One  possible  way  of  assigning  numbers  is  to  set  the  utility  of 
the  least  desired  level  to  0  and  set  u ( y )  -  u(x)  to  k  when  this 
difference  is  the  ktl1  one  in  the  ranking.  For  the  example 
above  this  implies 

u(a)  =  0, 

u(b)  -  u(a)  =3,  u  (b)  =  3, 

u(c)  -  u(b)  =  2,  u  (c)  =  5, 

u(d)  -  u(c)  =  1 ,  .  u  (d)  =  6. 

4 

* 

* 
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Successive  comparison  methods  work  with  groups  of 
utility  adjacent  criterion  levels,  and  attempt  to  assign  util¬ 
ities  to  them  by  examining  inequalities  between  sums  of  utilities. 
For  example,  suppose  again  u  (a)  <  u(b)<  u(c)  <  u(d).  Compare 
u(d)  with  u ( b )  +  u(c).  If  u(d)  <  u(b)  +  u(c),  compare  u(c) 
with  u(a)  +  u(b).  If  u(d)  >  u(b)  +  u(c)  compare  u(d)  with 
u ( a )  +u(b)  +  u(c).  Continue  in  this  manner  until  a  complete 
set  of  inequalities  is  obtained.  Numerical  utilities  may  now 
be  assigned  to  each  criterion  level  consistent  with  these  in¬ 
equalities.  Many  variants  of  this  method  are  possible. 

There  are  several  useful  types  of  trade-off  methods. 

The  single  trade-off  method  uses  a  set  of  indifference  curves 
between  two  criteria,  and  the  previously  derived  utility  func¬ 
tion  for  one  of  the  criteria  in  order  to  derive  the  utility 
function  for  the  other.  If  (ti,Si)  is  on  the  same  indifference 
curve  as  (t2,s2)  then  u ( 1 1 )  +  u(sj)  =  u(t2)  +  u(s2).  If  u(t.) 
is  known,  a  set  of  such  equations  can  be  solved  for  u(s.j)  by 
setting  u(s.j)  for  some  i,  to  an  arbitrary  value.  One  possi¬ 
bility  would  be  to  set  u(s.„)  to  0,  where  sm.  is  the  least  pre- 
f erred  s  1 evel  . 

Directly  assessed  preference  techniques  are  effective 
in  determining  utility  functions,  however  the  conditions  under 
which  the  additivity  assumption  holds  are  fairly  restrictive 
and  may  not  be  satisfied  in  all  project  selection  decision 
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2.4.2  Inferred  Preference  Techniques 

Inferred  preference  methods  attempt  to  deduce  the 
decision  maker's  utility  function  from  choices  he  makes.  These 
choices  may  be  actual  past  decisions  in  similar  situations  or 
they  may  be  decisions  made  on  simulated  problems  presented  to 
him.  Some  hypothesis  must  now  be  made  about  the  form  of  the 
utility  function.  If  it  is  assumed  to  be  linear  or  "quasi-linear" 
then  the  coefficients  are  estimated  using  standard  linear  re¬ 
gression  techniques.  If  the  utility  function  is  thought  to 
be  non-linear  and  involve  interactions  among  the  attributes, 
then  analysis  of  variance  is  used  to  determine  the  utility 
function . 

Inferred  preference  techniques  have  been  particularly 
successful  in  predicting  decisions.  Huber,  Sahney  and  Ford 
[49]  have  used  the  technique  to  model  professionals  judging 
the  quality  of  hospital  wards.  Huber,  Daneshgar  and  Ford  [48] 
have  applied  the  techniques  to  job  selection  decisions.  Both 
studies  used  several  types  of  functional  forms  and  found  that 
the  linear  one  had  the  greatest  predictive  power.  Einhorn  [27] 
used  a  linear,  and  two  "quas i -1 i near "  functions  to  model  job 
selection  decisions  and  graduate  school  admission  decisions. 

He  found  that  the  conjunctive  form  was  most  predictive  in  the 


A  quasi-linear  function  s  any  function  which  can  be 
made  linear  by  a  simple  transof rmat ion  For  example,  the  multi 
plicative  form  U  ■  tfX.  is  quasi-linear  since  log  U  *  E  log  X. 
i s  1 i nea  r .  1  1 


job  selection  case,  and  the  linear  function  was  most  applicable 
to  the  admission  decision  problem.  The  conjunctive  form  models 
the  strategy  which  rejects  alternatives  unless  they  exceed 
some  standard  with  respect  to  all  criteria.  If  even  one  criter¬ 
ion  level  is  less  than  the  standard  then  the  alternative  has  a 
low  utility  (i.e.  it  is  a  non-compensatory  strategy).  He  argues 
that  this  is  a  reasonable  strategy  in  situations  where  choosing 
the  wrong  alternative  would  be  very  costly  (such  as  job  selec¬ 
tion).  In  such  cases,  that  type  of  conservative,  non-compensa¬ 
tory  strategy  would  minimize  the  chances  of  costly  mistakes. 
Goldberg  [39]  used  the  same  functions  plus  two  more  "quasi- 
linear"  functions  to  model  psychol ogists  judgi ng  (MMPI)  test 
profiles.  He  concluded  that  the  linear  function  best  predicted 
decisions.  Hoffman  and  Wiggins  [47]  studied  the  same  problem 
using  a  linear  function,  a  quadratic  function,  and  a  sign  model 
(a  linear  combination  of  scores  and  functions  of  scores).  There 
was  some  evidence  that  interactions  among  criteria  affected 
decisions  (conf igural i ty) ,  but  even  so  the  linear  function  pre¬ 
dicted  decisions  well. 

Several  authors  have  suggested  using  inferred  prefer¬ 
ence  techniques  to  develop  a  utility  function  which  would  then 
be  used  as  a  decision  making  tool  to  help  eliminate  costly  in¬ 
consistencies.  Bowman  [10]  and  Kunreuther  [53]  have  used  the 

4 

techniques  in  several  managerial  dgcision  making  situations, 
ranging  from  inventory  policy  and  production  scheduling  decisions, 
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to  equipment  replacement  policy  decisions.  They  claim  that 
mistakes  due  to  a  manager's  inconsistency  are  more  costly  than 
mistakes  caused  by  his  misconception  of  the  problem.  Their 
experience  has  indicated  that  most  managers  have  a  good  under¬ 
standing  of  the  problems  that  face  them,  and  from  similar  past 
situations  are  aware  of  what  factors  and  indicators  are  most 
important.  In  several  cases  they  have  been  able  to  show  that 
this  type  of  analysis  of  past  decisions  has  led  to  a  decision¬ 
making  procedure  which  attained  better  results  than  decisions 
made  by  following  a  policy  developed  from  a  more  analytical 
study  of  the  situation. 

Yntema  and  Torgerson  [84]  have  suggested  a  computer- 
aided  decision  system  which  would  reduce  the  amount  of  time 
spent  on  routine  decisions  as  well  as  help  eliminate  inconsis¬ 
tencies.  They  use  an  analysis  of  variance  technique  to  derive 
the  decision  maker's  utility  function  which  is  then  used  as  the 
decision  rule  by  the  computer.  They  found  that  "main  effects" 

(the  linear  portion  of  the  function)  provided  an  excellent 
approximation  to  the  utility  function  if  utility  was  monotoni cal ly 
increasing  in  all  criteria,  i.e.  a  higher  criterion  level  is 
always  preferred  to  a  lower  one. 

In  all  of  the  studies  mentioned  the  decisions  predicted 
with  a  linear  utility  function  correlated  highly  with  actual 

4 

decisions.  This  is  not  meant  to  sj,igg'est  that  the  human  decision 
process  is  necessarily  linear.  However  a  linear  model  has  been 
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shown  to  be  an  excellent  "paramorphic  representation"  [18]  of 
decision  makers  in  many  situations.  That  is,  while  the 
decision  process  may  not  be  linear,  the  general  linear  model 
is  powerful  enough  to  reproduce  most  decisions  with  very  little 
error.  Goldberg  [39]  has  found  that 

.  .  .  for  a  number  of  different  judgment 

tasks  and  across  a  considerable  range  of 
judges ,  the  simple  linear  model  appeared 
to  characterize  quite  adequately  the  judg¬ 
mental  processes  involved  -  in  spite  of  the 
reports  of  the  judges  that  they  were  using 
cues  in  a  highly  configural  manner.  .  .  . 

Consequently  if  one’s  purpose  is  to  repro¬ 
duce  the  responses  of  most  judges,  then  a 
simple  linear  model  will  normally  permit 
the  reproduction  of  90-100 %  of  their 
reliable  judgment  variance,  probably  in 
most  -  if  not  all  -  clinical  judgment 
tasks. 

Since  our  purpose  in  deriving  a  utility  function  to 
help  analyze  the  project  selection  problem  is  to  reproduce 
decisions,  it  would  seem  appropriate  to  use  an  inferred  prefer¬ 
ence  technique.  From  the  studies  cited  one  would  expect  a 
linear  function  to  predict  decisions  well.  However,  to  allow 
more  flexibility  to  model  people  and  situations  where  it  is  not 
the  most  appropriate  form,  one  might  wish  to  consider  some  of 
the  non-linear  functions  as  well.  The  functions  which  are 
included  in  the  Interactive  Utility  Assessment  Procedure  (IUAP) 

developed  are  the  linear,  conjunctive,  disjunctive,  logarithmic 

« 

and  exponential  functions,  and  a  combination  of  the  conjunctive 
and  disjunctive  functions. 
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The  linear  function  is  of  the  following  form: 


U  =  l 


k=  1 


H  Xk 


where  U  is  the  utility  of  an  alternative,  Xk  is  the  level  of 

criterion  k  achieved  by  the  alternative,  and  ak  is  the  coefficent 

found  by  the  linear  regression  procedure.  In  many  cases  it  seems 

to  be  true  that  while  utility  is  monotoni cal ly  increasing  with 

respect  to  each  criterion,  at  some  point  the  incremental  utility 

of  a  unit  increase  in  criterion  level  decreases.  In  other 

words  the  utility  function  exhibits  decreasing  returns.  The 

simplest  method  of  modelling  the  decreasing  returns  characteristic 

is  with  a  piecewise  linear  concave  function.  In  that  case  the 

utility  of  a  unit  of  criterion  k,  up  to  the  point  Bk  is  a^. 

When  the  criterion  level  is  greater  than  B.  ,  each  additional 
?.  121 

unit  is  worth  ak  rather  than  ak,  and  ak  <  ak.  Thus  the  utility 
function  would  be  as  follows: 


U 


nn  n 


The  linear  forms  are  compensatory  in  that  a  high  level  of  one 
criterion  can  make  up  for  a  low  level  of  another. 

The  conjunctive  utility  function  used  by  Goldberg  [39] 
was  originally  proposed  by  Einhorn  E27,]: 
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U 


n 

=  n 

k=  1 


where  the  a^  are  coefficients  obtained  by  applying  linear  re¬ 
gression  to  the  equation 


n 

log  U  =  l  a k  log  X, 
k=  1 


In  one  dimension  the  function  is  a  parabola  (see  Figure  2.2(a)). 

A  low  level  of  any  criterion  would  reduce  the  utility  of  the  alt¬ 
ernative  regardless  of  how  high  the  other  criterion  levels  might  be 
The  disjunctive  function,  also  originally  proposed 
by  Einhorn,  models  the  strategy  which  chooses  alternatives  on 
the  basis  of  their  "best"  criterion  level.  The  general  form  is 


n 

u  *  n 

k-1 


where  d^  is  the  maximum  possible  value  of  criterion  k.  The  co¬ 
efficients  ak  are  determined  by  applying  linear  regression  to 


log  U 


k-1 


log  J< 


k’ 
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In  one  dimension  the  function  is  a  hyperbola  (see  Figure  2.3 
(a)).  The  closer  a  criterion  level  is  to  its  maximum  value  d^, 
the  larger  the  fraction  l/Cd^-X^)  and  the  larger  the  utility 
assigned  to  the  alternative.  When  the  criterion  level  is  high 
the  function  provides  a  great  deal  of  discriminatory  power 
(i.e.  the  derivative  is  large).  This  corresponds  to  choosing 
between  alternatives  on  the  basis  of  their  best  criterion  level. 
The  disjunctive  form  is  also  non-compensatory .  If  an  alterna¬ 
tive's  best  criterion  level  is  only  average,  then  similar  average 
levels  of  other  criteria  will  not  make  it  more  attractive  than 
an  alternative  which  has  one  very  high  score  and  the  rest  very 
low. 

The  next  two  models,  the  logarithmic  and  the  exponen¬ 
tial  were  suggested  by  Goldberg  [39]  to  determine  whether  a 
logarithmic  transformation  of  the  criterion  levels  (logarithmic), 
or  the  value  judgements  (exponential)  would  provide  a  better 
fit  to  the  observed  utility  function.  The  logarithmic  form  is 

n  n  ®  k  ^  k 

U  *  y  a,,  log  X.  ,  and  the  exponential  form  is  U  =  II  e  , 

k-l  K  K  k»  l 

n 

or  log  U  =  l  a.X.  . 

k-l  K  K 

Another  non-linear  function  to  consider  is  a  combina¬ 
tion  of  the  conjunctive  and  disjunctive  forms: 

y 

* 
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or 

log  U  =  j  ak  log  Xk  -  log  (dk  -  Xfc)J  , 

where  dk  is  the  maximum  possible  value  of  criterion  k  and  the  ak 
are  coefficients  found  by  applying  linear  regression  to  the 
second  form.  Alternatives  which  have  a  high  level  with  respect 
to  one  criteria  are  given  a  high  utility  since  Xk  and  1 / d k  -  Xk 
would  both  be  large.  However,  none  of  the  other  criteria  can 
be  too  low  or  the  utility  would  be  reduced. 

In  the  Interactive  Utility  Assessment  Procedure  the 
coefficients  for  all  of  the  forms  are  obtained  by  applying  a 
linear  regression  technique  to  the  results  of  a  set  of  decisions. 
The  decision  maker  is  first  as-ked  to  specify  the  lowest  and 
highest  possible  values  of  each  of  the  criteria  he  wishes  to 
consider.  This  information  is  used  to  randomly  generate  alter¬ 
natives  (setsof  criterion  levels).  The  decision  maker  is  then 
interactively  asked  to  rate  the  alternatives  on  an  arbitrary 
scale  from  1-100.  One  is  the  utility  of  an  alternative  which 
has  the  lowest  possible  values  of  all  criteria,  and  one  hundred 
is  the  utility  of  an  alternative  wi'th^the  highest  possible 
values  of  all  criteria.  These  ratfngs  reflect  the  decision  maker's 
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conception  of  the  difference  in  value  between  such  sets  of 
criterion  levels.  They  have  no  intrinsic  significance,  and 
are  meaningful  only  in  relation  to  one  another.  The  decision 
maker  could  just  as  well  have  used  a  100-300  scale  to  rate  the 
al ternati ves . 

The  alternatives  are  rated  one  at  a  time  as  they  are 
generated.  The  decision  maker  is  then  presented  with  the  entire 
set  of  alternatives  and  may  adjust  his  ratings.  A  consistency 
check  is  applied  to  the  set  of  ratings.  Any  alternative  which 
dominates  another  (has  higher  levels  on  all  criteria),  and  yet 
has  a  lower  rating,  is  pointed  out,  and  the  decision  maker  is 
asked  to  re-evaluate  it.  Once  the  ratings  are  internally  con¬ 
sistent  i.e.  satisfy  the  transitivity  assumption,  and  the  decision 
maker  is  satisfied  with  them,  they  are  used  as  the  dependent  var¬ 
iable  in  a  linear  regression  procedure,  and  the  coefficients  for 
the  various  tynes  of  utility  functions  are  determined. 

Slovic  and  Lichtenstein  [71]  discuss  the  problems  that 
can  arise  when  the  alternatives  used  in  the  decision  situations 
posed  to  derive  the  utility  function  have  randomly  generated 
criteria  levels.  If  the  assigned  criteria  levels  are  unrealistic 
or  outside  the  range  the  decision  maker  is  accustomed  to  handl¬ 
ing,  then  he  may  not  be  able  to  accurately  rate  the  alternatives. 
However,  the  technique  used  in  the  IUAP  program,  assigns  random 
values  between  the  maximum  and  minimufo  possible  values  of  each 
criterion.  In  that  way  each  value  is  realistic  and  the  problem 
does  not  arise. 
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Another  problem  occurs  when  the  decision  criteria  are 
not  independent.  In  that  case  randomly  generated  criteria 
levels  could  well  violate  expected  relations  between  criteria. 
For  example,  if  two  of  the  criteria  were  the  amount  of  energy 
consumed  and  the  increace  in  the  pollution  level,  one  would 
expect  a  high  level  of  one  criterion  to  be  associated  with  a 
high  level  of  the  other.  The  random  generation  technique  used 
here  would  not  guarantee  that.  In  situations  where  the  expected 
relationship  is  not  observed,  the  decision  maker  essentially 
disregards  one  of  the  conflicting  criteria.  In  cases  where 
other  criteria  support  the  hypothesis  that  one  of  the  criterion 
levels  is  wrong,  this  is  a  reasonable  procedure.  However  at 
other  times  it  leads  to  incons i stency  and  unrelability.  One 
method  of  obviating  the  problem  would  be  to  employ  a  rejection 
technique.  The  generated  alternatives  could  be  examined  before 
being  presented  to  the  decision  maker,  and  rejected  if  the 
criterion  levels  violate  the  expected  relationships. 

Another  cause  of  incons i s tency  is  the  availability 

of  a  large  number  of  criteria.  Einhorn  [27]  has  shown  that  as 

the  number  of  criteria  increases,  consistency  decreases.  One 

possible  explanation  of  this  phenomenon  is  that  when  the  number 

of  criteria  to  be  considered  exceeds  the  decision  maker's 

information  processing  capacity,  he  chooses  only  a  subset  of 

them  on  which  to  base  his  judgment.  This  subset,  however,  is 

not  the  same  each  time.  This  hypothesis  is  strengthened  by 

* 

the  fact  that  the  number  of  criteria  that  are  statistical ly 
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significant  in  decisions  is  generally  smaller  than  the  decision 
maker  would  estimate.  He  may  remember  using  them  all  at  one 
time  or  another,  but  the  more  important  ones  dominate.  Slovic 
and  Lichtenstein  feel  that  the  most  important  criterion  generally 
accounts  for  40%  of  a  decision  maker's  predictable  variance, 
and  the  three  most  important  account  for  80%.  The  number  of 
criteria  that  may  be  used  without  causing  serious  inconsistencies 
appears  to  depend  a  great  deal  on  the  problem  being  considered 
and  the  relationship  between  the  criteria.  Whereas  only  the 
most  important  4  or  5  are  generally  statistically  significant, 

10  are  allowed  in  the  IUAP  program  and  could  probably  be  con¬ 
sidered  before  the  decision  maker  would  be  faced  with  an 
"information  overload"  which  is  assumed  to  lead  to  inconsistency. 

Of  the  many  methods  for  handling  multiple  criteria  . 
considered,  the  inferred  preference  technique  for  deriving 
utility  functions  used  by  the  IUAP  program  seems  to  be  the  most 
appropriate  in  analyzing  the  project  selection  problem.  Most 
of  the  sequential  elimination  methods  reduce  the  set  of  possible 
best  alternatives,  but  the  remaining  ones  must  still  be  ordered 
using  some  Other  method.  The  restrictive  logic  of  several  of 
the  methods  and  their  non-compensatory  nature  make  them  inappro¬ 
priate  in  many  cases.  They  are  heuristic  methods  of  handling 
multiple-criteria  decision  situations  which  are  not  based  on  a 
logical  strategy.  The  spatial  proximity  methods  are  computa¬ 
tionally  more  complex  than  the  inferred  preference  technique. 

V» 

They  are  also  more  difficult  for  the  decision  maker  to  under- 
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stand  and  use.  Therefore,  from  a  practical  point  of  view,  the 
inferred  preference  technique  is  more  suitable.  The  mathematical 
programming  methods  are  applicable  to  the  project  selection 
problem  since  it  has  continuous  criterion  levels.  They  have 
been  successful  in  solving  several  real-world  problems  and  might 
prove  effective  in  solving  the  project  selection  problem  as 
well.  They  are  more  difficult  to  implement  than  the  inferred 
preference  technique  however,  and  have  not  been  used  for  that 
reason.  The  directly  assessed  preference  techniques  for  deriving 
utility  functions  are  based  on  the  assumption  that  the  utility 
functions  for  each  criteria  are  additive.  The  conditions  under 
which  this  additivity  assumption  holds  may  not  be  met  in  many 
project  selection  decision  situations  and  thus  these  methods 
are  not  always  appropriate.  Inferred  preference  techniques 
such  as  IUAP  have  been  shown  to  be  very  successful  in  modeling 
decision  makers  and  predicting  their  preferences.  Since  this 
modeling  capability  is  the  major  goal  in  developing  a  project 
selection  algorithm,  this  technique  seems  to  be  the  appropriate 
one  to  use. 


Chapter  3 


A  NEW  PROJECT  SELECTION  ALGORITHM 


In  any  rational  decision  process  it  is  necessary  to 
evaluate  outcomes  in  terms  of  goals.  In  the  last  chapter  it 
was  suggested  that  the  most  appropriate  evaluation  method  for 
the  project  selection  problem  is  to  derive  a  measure  of  the 
utility  of  each  alternative  as  a  function  of  its  criterion 
levels.  The  most  efficient  and  practical  method  of  deriving 
a  utility  function  in  this  case  is  the  inferred  preference 
technique.  This  technique  derives  a  utility  function  by  asking 
the  decision  maker  to  reveal  his  preferences  through  choice. 
There  are  several  forms  this  utility  function  may  take  but 
many  studies  suggest  that  a  linear  form  often  predicts  decisions 
best.  Therefore,  the  linear  form,  and  a  modification  of  it, 
the  piecewise  linear  form,  were  selected  for  the  new  decision 
algori  thin. 

3.1  DERIVING  A  LINEAR  UTILITY  FUNCTION 

The  Interactive  Utility  Assessment  Procedure  (IUAP) 

V. 

for  identifying  a  utility  function  is  used  in  the  project 
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selection  algorithm^  to  derive  the  linear  form.  The  decision 
maker  is  first  asked  by  a  conversational  subroutine  to  describe 
his  decision  problem.  How  many  projects  are  being  considered? 

How  many  criteria  are  relevant?  What  is  the  suggested  or  requested 
level  of  funding  for  each  project,  and  what  is  the  maximum  and 
minimum  funding  level  possible?  What  level  of  each  criterion  would 
each  project  achieve  (assuming  it  is  successful  and  funded  at  the 
requested  level)?  This  information  is  used  to  determine  the 
maximum  and  minimum  value  of  each  criterion,  Xm^x  and  Xm^n. 

Then,  sets  of  criterion  levels  between  these  bounds  are  randomly 
generated.  These  sets  of  criterion  levels  can  be  thought  of 
as  "pseudo-projects."  Since  each  project  can  be  funded  at  an 
infinite  number  of  levels  between  its  upper  and  lower  funding 
bounds,  a  combination  of  projects  funded  with  various  amounts 
might  result  in  a  set  of  criterion  levels  with  the  same  values 
as  the  pseudo-project.  Each  pseudo-project  is  therefore  rep¬ 
resentative  of  a  possible  research  and  development  program 
(portfolio  of  projects).  At  this  point  in  the  algorithm,  IUAP 
is  used  to  interactively  illicit  preference  ratings  and  deter¬ 
mine  regression  coefficients  for  the  linear  form. 

The  next  step  in  the  procedure  is  to  determine  how 
well  the  linear  model  predicts  the  decision  maker's  ratings. 

Three  different  coefficients  of  the  correlation  between  the 

Figure  3.1  is  a  flow  chfcrt  of  the  algorithm. 
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actual  and  predicted  ratings  are  calculated  by  the  program; 
the  Pearson  or  product  moment  correlation,  the  Spearman  or  rank 
correlation,  and  a  di s tance-error  correlation.  Since  these 
correlations  are  calculated  on  the  same  data  used  to  derive  the 
regression  coefficients  one  would  expect  them  to  be  high.  At 
this  point  a  new  set  of  pseudo-projects  could  be  generated  and 
rated.  The  correlations  between  the  actual  and  predicted  ratings 
of  this  set  would  be  a  better  indicator  of  the  goodness  of  fit 
of  the  linear  model.  However,  another  set  of  ratings  for  this 
purpose  seems  to  place  an  unnecessary  burden  on  the  decision 
maker.  A  correlation  level  of  0.9  was  arbitrarily  chosen  as  a 
standard.  If  any  of  the  correlations  are  greater  than  0.9  then 
the  linear  model  will  probably  fit  well  enough  to  be  useful. 
However,  the  decision  maker  is  given  the  option  to  continue  the 
process  in  the  hope  of  finding  a  better  fitting  utility  function. 
If  none  of  the  correlations  are  greater  than  0.9,  or  if  the 
decision  maker  exercises  his  option  to  continue,  then  a  piece- 
wise  linear  utility  function  is  derived  in  an  attempt  to  improve 
the  correspondence  between  choices  and  predictions. 

3.2  DERIVING  A  PIECEWISE  LINEAR  UTILITY  FUNCTION 

If  the  decision  maker's  utility  function  is  best 
approximated  by  a  piecewise  linear  .function,  we  would  expect  the 

4 

* 
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conf i gura t i on  of  pseudo-projects  to  resemble  one  of  the  cases 
depicted  in  Figure  3.2.  The  pseudo-project  with  the  most 
poorly  predicted  score  is  found  by  computing  the  absolute  value 
of  the  difference  between  the  predicted  score  and  the  actual 
score  for  all  pseudo-projects,  and  identifying  the  largest 
such  difference  d.  To  eliminate  the  possibility  that  this 
deviation  is  simply  an  error  on  the  part  of  the  decision  maker, 
he  is  asked  to  rate  another  pseudo-project  in  the  same  neighbour¬ 
hood  as  the  worst  point.  The  criterion  levels  of  this  new  point 
are  set  at  randomly  generated  values  between  the  two  adjacent 
criterion  levels.  This  new  pseudo-project  replaces  the  most 
poorly  fitted  one  and  the  linear  regressior  procedure  is  re¬ 
applied  using  the  new  set  of  ratings  as  dependent  variables. 

Once  again  the  pseudo-project  with  the  more  poorly  predicted 
score  is  found.  This  cycle  is  repeated  until  one  of  three 


2 

The  adjacent  criterion  levels  are  found  by  examining 
each  pseudo-project  to  find  the  greatest  criterion  level  less 
than  the  corresponding  one  for  the  worst  point,  and  the  smallest 
criterion  level  greater  than  the  corresponding  one  for  the  worst 
point,  for  all  criteria.  Another  method  of  generating  a  new 
pseudo-projec t  in  the  neighbourhood  of  the  worst  point  would 
be  to  consider  a  circle  of  radius  r  about  the  point.  Criterion 
level  k  of  the  new  point  would  then  be  generated  at  random 
from  the  interval  (Xk  -  r,  +  r),  where  Xk  is  the  criterion 
level  of  the  worst  point. 

4 

n 

4 
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things  happens.  1.)  The  last  worst  point  found  is  near 
one  of  the  previous  worst  points.  In  that  case  an  area  has 
been  identified  where  the  linear  form  does  not  predict  well, 
and  one  might  suspect  that  a  piecewise  linear  function  would 
be  more  appropriate.  2.)  The  correlation  between  the  actual 
and  predicted  ratings  improves  enough  that  the  decision  maker 
decides  the  linear  form  is  adequate.  3.)  The  iteration 
counter  k  becomes  greater  than  max  (3,  4m/5),  where  m  is  the 
number  of  criteria  being  considered.  In  this  case,  after 
a  "reasonable"  number  of  iterations,  no  area  has  been  found 
where  the  linear  form  consistently  predicts  ratings  poorly. 

One  may  conclude  therefore,  that  the  decision  maker's  underly¬ 
ing  utility  function  is  not  piecewise  linear  and  that  a  piece- 
wise  linear  representation  of  it  would  not  likely  predict 
ratings  better  than  the  linear  form. 

If  case  1.)  occurs  then  the  next  step  in  the  algor¬ 
ithm  is  to  find  the  breakpoints  Bk  of  the  piecewise  linear 
form 

m 

U  =  £  [a2  Xk  +  ( a k  -  ak)min(Xk,Bk)]. 

k  —  1  k 

The  breakpoints  are  where  the  phenomenon  of  decreasing  returns 


The  new  worst  point  is 
all  of  its  criterion  levels  fall 
/vmax  vmin)  /vmax 


near  the  last  worst  point  if 
in  the  interval 
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where 


criterion  level  of  the  old  worst  point  and  N  is  the  total  number 
of  pseudo-projects  generated. 
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occurs,  i.e.  where  the  marginal  value  of  a  unit  increase  in 

the  criterion  level  decreases.  For  criterion  levels  less  than 

the  breakpoint,  the  added  value  of  one  unit  of  criterion  k 
1 

is  a^.  When  the  criterion  level  is  greater  than  B^,  the  added 

2  2  1 

value  of  one  unit  is  a^  and  afc  <  a^. 

One  possible  method  of  obtaining  the  breakpoints 

is  to  ask  the  decision  maker  for  them.  He  may  be  aware  of 

a  discontinuity  in  his  utility  function  with  respect  to  one 

or  more  criteria,  and  may  be  able  to  indicate  where  it  occurs. 

In  most  instances  however,  this  will  not  be  the  case.  The 

computerized  procedure  finds  the  breakpoints  by  examining  the 

last  worst  point  found.  If  all  of  its  criterion  levels  are  in 

the  middle  half  of  the  range  of  possible  values  of  that  criterion, 

2  5  7  5 

i.e.  between  and  in  Figure  3.2,  then  case  (a)  in  Fig¬ 

ure  3.2  is  applicable.  The  criterion  levels  of  the  point  are 
then  taken  as  the  breakpoints  for  each  criterion.  If  all 
of  the  criterion  levels  of  this  worst  point  are  not  in  the 
middle  half  of  the  range  of  possible  values  of  the  criterion, 
then  case  (b)  or  (c)  of  Figure  3.2  may  apply.  In  this  case  the 
criterion  levels  of  this  point  cannot  be  used  as  the  breakpoints. 

Some  other  poorly  fitted  pseudo-project  with  criterion  levels 
closer  to  the  middle  of  the  range  must  be  found.  The  pseudo¬ 
projects  are  searching  in  decreasing  order  of  d  (distance  be¬ 
tween  actual  and  predicted  ratings)  until  one  satisfying  the 

V 

"middle  of  the  range"  rule  is  found  (^in  which  case  its  criterion  <. 
levels  are  used  as  the  breakpoints),  or  until  half  of  the 


Figure  3.2  Piecewise  Linear  Utility  Functions 

pseudo-projects  have  been  examined.  If  the  most  poorly  predicted  half 
of  the  pseudo-projects  does  not  contain  a  point  in  the  middle  of 
the  range,-  then  the  distribution  of  sample  points  does  not  resemble 
case  (b)  or  (c)  of  Figure  3.2,  and  it-Js  unlikely  that  a  piecewise 
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linear  form  would  be  a  more  accurate  representation  of  the 
decision  maker's  actual  utility  function,  than  the  linear 
form. 


Once  the  breakpoints  of  the  piecewise  linear  form 

have  been  determined,  the  next  step  is  to  find  the  coefficients 
l  2 

ak  and  ak  ,  k=l,...m.  This  is  done  by  applying  regression  to 


U 


l  l  2 

ak  Xk  +  ak 


l 

where 


=  min  (X^,  8^)  and 
2 

Xk  ”  mdx 


0). 


Since  twice  as  many  coefficients  as  before  are  being  calculated, 

another  4m  pseudo-projects  are  generated  and  interactively 

rated  before  the  regression  technique  is  applied. 

The  next  step  in  the  procedure  is  to  determine  which 

of  the  two  forms  (the  linear  or  the  piecewise  linear)  fits 

best.  The  correlation  coefficients  between  the  actual  ratings 

given  the  original  set  of  pseudo-projects  by  the  decision 

maker,  and  the  ratings  predicted  by  the  piecewise  linear  form 

are  calculated.  These  correlation  coefficients  are  compared 

with  the  corresponding  ones  for  the  linear  form.  The  linear 

form  has  an  unfair  advantage  in  this  comparison  since  its 

correlation  coefficients  are  calculated  on  the  same  data  as 

was  used  to  derive  its  regression  coefficients.  Here  again 

* 

another  set  of  pseudo-projects  could  be  generated  and  used 
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to  test  the  two  models  more  fairly.  But  the  imposition  of 
a  third  set  of  ratings  was  found  to  be  unacceptable  to  many 
decision  makers.  If  any  of  the  correlation  coefficients  for 
the  piecewise  linear  form  are  greater  than  the  corresponding 
ones  for  the  linear  form,  despite  the  bias  in  favour  of  the 
linear  form,  then  the  piecewise  linear  form  would  seem  to  fit 
best.  If  all  of  the  coefficients  for  the  linear  form  are 
greater  than  the  ones  for  the  piecewise  linear  form,  then  the 
linear  form  may  or  may  not  provide  a  better  fit.  However, 
for  lack  of  another  method  of  measuring  the  fit  of  the  two 
forms,  the  linear  one  would  be  chosen  in  that  case.  If  the 
decision  maker  is  still  not  satisfied  with  the  fit  of  the  chosen 
utility  function  the  whole  procedure  could  be  repeated.  This 
time  more  pseudo-projects  would  be  generated  at  each  stage 
in  order  to  find  a  better  fitting  utility  function. 

The  question  of  how  many  pseudo-projects  to  generate 

at  each  stage  is  an  important  one.  The  more  ratings  available 

the  better  the  fit  of  the  utility  function.  However,  if  too 

many  ratings  are  required,  the  decision  maker  will  become 

tired  and/or  annoyed  with  a  resulting  decline  in  accuracy. 

There  must  be  an  optimum  number  of  pseudo-projects  to  generate 

which  would  minimize  these  two  types  of  problems.  Clearly 

this  number  should  be  related  in  some  way  to  the  number  of 

criteria  used.  As  the  number  of  criteria  (and,  in  a  sense, 

« 

degrees  of  freedom  in  the  utility  function)  increase,  more 
points  will  be  needed  for  a  good  fit.  The  number  of  pseudo¬ 
projects  to  generate  was  set  at  4m,  when  m  is  the  number  of 
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criteria.  If  5  criteria  were  considered  relevant,  which  is 
more  than  the  number  most  studies  found  to  be  statistically 
significant  in  influencing  ratings,  then  20  pseudo-projects 
would  be  generated.  Twenty  was  found  to  be  a  reasonably  large 
number  of  ratings  with  which  most  decision  makers  can  cope. 

The  number  of  criteria  is  certainly  not  limited  to  5,  however. 

If  the  decision  maker  wishes  to  use  more,  then  he  must  rate 
a  correspondingly  larger  number  of  pseudo-projects. 

When  the  piecewise  linear  form  is  being  derived, 
another  set  of  4m  pseudo-projects  is  generated.  The  question 
arises  as  to  whether  the  decision  maker  can  rate  this  new  set 
of  pseudo-projects  so  that  the  ratings  are  consistent  with 
the  ones  for  the  previous  set.  Since  the  type  of  decision  maker 
this  procedure  is  intended  to  model  is  one  who  is  familiar 
enough  with  his  problem  and  his  preferences  so  that  his  view¬ 
point  would  not  change  as  a  consequence  of  the  exercise,  he 
is  considered  to  be  capable  of  consistency.  As  an  aide  to 
achieving  consistency  the  previous  set  of  pseudo-projects  and 

ratings  are  presented  to  him  as  reference  points. 

2 

A  third  question  which  arises  is  why  ak  should  be 

less  than  aj^.  The  regression  procedure  certainly  does  not 

2 

guarantee  this.  If  in  fact  it  turns  out  that  a^  is  greater 
than  a£,  then  the  decision  maker's  responses  do  not  satisfy 
the  assumption  of  decreasing  returns  with  respect  to  criterion 
k  (see  Figure  3.3).  Since  our  solution  procedure  depends  on 
concavity,  and  it  would  seem  unlikely  that  a  decision  maker's 


utility  function  should  exhibit  increasing  returns  when  dealing 
with  real  problems  if  this  situation  occurs,  then  the  piece- 
wise  linear  form  is  discarded  and  the  linear  form  chosen. 


Another  method  of  dealing  with  the  problem  would 
be  to  re-apply  the  regression  procedure  to  a  new  form  which 
is  piecewise  linear  with  respect  to  only  those  criteria  which 
satisfy  the  decreasing  returns  assumption,  i.e. 
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where  D  is  the  set  of  those  criteria  which  exhibit  decreasing 
returns . 


3.3  FORMULATING  THE  OBJECTIVE  FUNCTION 

Once  the  appropriate  utility  function  has  been  chosen, 
it  can  be  combined  with  the  decision  maker's  knowledge  of  the 

criterion  levels  of  each  of  the  projects  under  consideration, 
to  construct  an  objective  function  for  use  in  a  mathematical 
programming  algorithm.  Consider  first  the  linear  form.  The 

total  utility  of  any  portfolio  of  projects  is 
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where  Xk^  is  the  level  of  criterion  k  that  project  j  actually 
achieves.  X^  varies  with  the  amount  of  funds  allocated  to 
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project  j  and  cannot  be  known  exactly  until  after  the  project 

is  completed.  However,  the  decision  maker  has  some  subjective 
estimate  of  the  probability  that  project  j  will  be  successful 
when  funded  at  various  levels.  The  expected  value  of  7.  .  is 

K  J 

then  XkjPj  ( x  j ) ,  where  is  the  level  of  criterion  k  that 
the  decision  maker  specifies  in  his  description  of  the  project, 
and  P.  (x.)  is  the  probability  that  project  j  will  succeed 

J  J 

when  funded  at  x^. .  The  expected  utility  of  any  research  and 
development  program  is  then 
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Souder  [77,78]  has  studied  the  effect  on  resulting 
decisions  of  three  different  forms  of  the  P-(x.)  function. 

J  J 

The  three  forms  he  chose  were  a  linear  function,  a  piecewise 
linear  function  and  a  non-linear  function  (either  exponential 
or  S-shaped).  These  functions  were  derived  by  fitting  curves 
through  points  specified  by  the  decision  maker  (see  Figure  3.4). 
He  found  that  the  linear  function  resulted  in  decisions  which 
maximized  profit,  but  that  the  piecewise  linear  function  was 
most  often  preferred  because  it  had  other  desirable  properties. 
"The  piecewise  model  was  more  frequently  preferred  than  the 
other  two  forms,  largely  because  of- its  ability  to  select 
compromise  portfolios.  The  'compromise'  portfolios  were  those 
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Non-linear  Functions'. 

Exponential  and  3-shapeti 

rtll  tnree  forms  are  derived  by  asklnpr  the  decision  maker  for 
the  probability  project  J  will  succeed  if  funded  -it  the  various 
levels. 

Figure  3. im¬ 
probability  of  Success  Functions 
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yielding  acceptable  anticipated  profits  while  still  maintain¬ 
ing  minimum  funding  on  imminent  failure  outcome  projects  and 
providing  a  balance  of  intrinsic  portfolio  attributes"  [78]. 

The  piecewise  linear  function  is  more  suitable  for  multiple 
criterion  situations  because  of  this  compromise  characteristic. 

In  that  phase  of  the  procedure  where  the  decision 
maker  is  asked  to  describe  the  projects  he  is  considering,  he 

must  specify  the  requested  or  suggested  level  of  funding  f., 

J 

and  the  upper  and  lower  funding  bounds  u.  and  £ .  respectively, 

J  J 

for  each  project  j.  These  bounds  may  be  set  by  organizational 
policy,  or  they  may  be  simply  limits  which  seem  reasonable  to 
the  decision  maker.  The  upper  bound  might  be  the  point  where 
the  decision  maker  feels  increased  funding  would  not  signifi¬ 
cantly  increase  the  probability  of  success.  The  lower  bound 
might  be  such  that  if  the  project  were  to  be  funded  at  an 
amount  below  it,  there  would  be  virtually  no  possibility  of 
success.  In  addition  to  specifying  these  funding  levels,  the 
decision  maker  must  estimate  the  probability  of  success  of  each 
project  when  funded  at  its  upper  and  lower  bounds.  The  prob¬ 
ability  of  success  of  project  j  when  funded  at  level  x.  ,  (x-  = 

J  J 

0,  or  £ •  5  x.  5  u.)  is  then 

J  J  J 


Pj(xj)  -  Vj  *  (a,  -  «j>  yj 
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where  P^x.)  is  the  probability  project  j  will  succeed  when  it 

J  J 

is  funded  at  £.,  6-  is  the  probability  it  will  succeed  when 

J  J 

funded  at  u.,  z.  is  a  0  or  1  variable  which  indicates  whether 

J  J 

or  not  project  j  is  funded  at  all  (i.e.  x.  is  at  least  l  ■ ) , 

J  J 

and  y.  is  a  variable  between  0  and  1  which  is  the  amount  of 

J 

funding  above  l.  allocated  to  project  j.  Using  this  notation, 
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Xj  =  £JZj  + 
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This  is  the  expression  which  will  be  used  as  the  objective 
function  in  the  mathematical  programming  algorithm. 


3.4  FORMULATING  THE  CONSTRAINTS 

The  next  step  is  to  determine  the  constraints  within 
which  the  decision  maker  must  work.  These  are  typically  con¬ 
straints  on  resources  such  as  money  and  manpower.  The  decision 
maker  must  specify  the  amount  of  each  resource  i  used  by  each 
project  j  when  funded  at  its  requested  funding  level  f . ,  denoted 

J 

by  S..,  and  the  total  amount  of  each  resource  available,  R. . 

1  J  *  1 

The  resource  constraints  may  then  be  formulated  as 
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I  x  <  B  ,  or  l 


j-l 


J-l  1 


£.2.  +  (u  .  -  l .)  y  . 
J  J  J  J  3 1 


<  B  and 


n  S 


n  S .  . 

_LL 


l  -}i  x  <  R  or  J  -W 
j-l  j  3  '  j-l  fj 


£.z.  +  (u  .  -  £.)  y. 
J  J  J  J 


1  Ri  Vi  , 


s  .  . 

where  — is  the  amount  of  resource  i  required  per  dollar  in¬ 
ti 

vested  in  project  j  and  B  is  the  total  budget. 

There  is  an  assumption  implicit  in  this  formulation 

of  the  resource  constraints.  The  amount  of  resource  i  required 

by  project  j  is  assumed  to  vary  linearly  with  the  amount  of 

funding  .  This  is  a  fairly  reasonable  assumption  for  most 

types  of  resources  when  x.  falls  in  the  range  [A.,u.].  This 

J  J  J 

range  is  where  changes  in  funding  result  in  significant  changes 

in  the  probability  of  success  and  thus  in  the  amount  of  each 

attribute  produced  and  each  resource  consumed.  Outside  of 

this  range  changes  in  funding  may  not  be  reflected  by  changes 

in  resource  requirements.  It  would  seem  reasonable  that  the 

consumption  of  resources  such  as  energy  and  manpower,  which 

are  essentially  continuous  (can  be  purchased  in  any  amounts) 

would  vary  linearly  with  funding  in  this  range.  However  there 

are  some  types  of  resources  which  are  zero-one  in  nature. 

For  example,  if  a  piece  of  elaborate  equipment  is  necessary 

« 

(such  as  a  computer  or  a  cyclotron)-,  three-quarters  of  the 
machine  would  not  be  particularly  useful.  In  cases  like  this 
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then,  the  assumption  is  not  valid.  Most  resources  considered 
in  project  selection  problems  however,  are  of  the  continuous 
type. 

Other  types  of  constraints  are  possible  as  well. 

There  may  be  relationships  among  the  projects  such  that  if  one 
project  or  group  of  projects  is  funded,  then  others  may  not 
be.  For  example,  a  government  agency  may  be  concerned  that 
no  more  than  N  projects  from  each  region  of  the  country  are 
funded.  In  that  case  the  necessary  set  of  constraints  are 

l  z  <  N  V  l  , 

jeJ*  J 

where  is  a  set  which  indexes  all  projects  submitted  from 
region  A.  Another  possible  relationship  among  project  occurs 
when  one  project  depends  on  results  from  another.  In  that 
case  the  dependent  project  A  should  not  be  funded  unless  the 
independent  project  B  is  also  funded.  The  constraint  z^  -  Zg 
will  insure  that  project  A  is  not  funded  (z^  *  0)  if  project  B 
is  not  funded  (zB  -  0).  Other  types  of  relationships  may  be 
envisaged  and  can  be  readily  included  in  the  constraint  set. 

Another  useful  type  of  constraint  would  be  to  specify 
minimum  levels  of  some  or  all  criteria  which  must  be  achieved 
by  the  portfolio  of  projects.  These  constraints  would  have 

4 

the  following  form:  T 
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n 

jf,  <“jzJ  +  (6j  -  XM  »  Mk- 

where  is  the  minimum  acceptable  level  of  criterion  k. 

A  government  agency  might  wish  to  define  such  a  set  of  min¬ 
imum  standards  in  order  to  ensure  approval  of  all  their  funding 
programs . 

3.5  THE  FORMULATION  WITH  A  LINEAR  UTILITY  FUNCTION 

The  complete  formulation  of  the  project  selection 
problem  with  a  linear  utility  function  is  then  as  follows: 


other  constraints 
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(4) 


Zj  =  0  or  1  ,  and'  0  <  <  1  Vj  (5) 


The  project  selection  problem  may  now  be  solved  using  a  mixed 
integer  branch  and  bound  algorithm. 


3.6  THE  FORMULATION  WITH  A  PIECEWISE  LINEAR  UTILITY  FUNCTION 

If  the  piecewise  linear  utility  function  is  chosen 
as  the  one  which  predicts  the  decision  maker's  ratings  best,  then 
the  objective  function  in  this  formulation  must  be  changed. 

In  that  case,  the  total  utility  of  any  portfolio  of  projects  is 

k  J,  *  K  -  ak)m1n 


level  of  criterion  k  actually  achieved  by  project  j.  Again  this 
may  be  translated  into  expected  utility  using  the  two  probability 
of  success  estimates  provided  by  the  decision  maker. 
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In  order  to  convert  this  objective  function  into  a 
useful  form,  two  new  sets  of  variables  are  required;  tk  and  s^. 
tk  is  the  total  expected  level  of  criterion  k  and  sk  is  the 
amount  by  which  this  level  exceeds  the  breakpoint  Bk,  if  any. 
Now  the  problem  may  be  formulated  as  follows: 


max  l  aktk  -  ak  -  ak  sk 


s-1-  *k  *  ]  (Vj  +  (8j  -  V  *j)  X; 


V  k  (1 ) 


h  5  ]  («j*j  *  (6j  •  V  J'j)  xjk  -  Bk  vk  (2) 


J,  *  (uj  -  V  yj  2  8 
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J,  T f  (Vj  +  <uj  -  V  *j)  5  R1  "  ‘4> 


y  .  <  z  . 
*2  J 


Vj  (5) 


other  constraints 


(6) 


Zj  =  0,  1  and  0  <  <  1  Vj  (7) 


*k  ’  sk  >  0 


Vk  (8) 


Constraints  ( 3 ) - ( 7 )  are  the  same  as  the  constraints  in  the 

previous  formulation.  Constraints  (1)  set  tk  to  the  expected 

level  of  criterion  k  and  constraints  (2)  set  sk  to  the  difference 

between  the  level  of  criterion  k  and  the  breakpoint  Bk.  If  this 

difference  is  negative,  then  sk  ■  0  by  (8).  sk  will  never  be 

assigned  a  value  greater  than  the  excess  of  criterion  level  k 

over  Bk  because  the  coefficient  of  sk  in  the  objective  function 

is  negative  (by  concavity  ak  >  ak)..  Therefore,  optimality  will 

* 

force  sk  to  be  as  small  a  non-negative  value  as  possible. 
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This  formulation  of  the  project  selection  problem 
can  also  be  solved  by  a  branch  and  bound  mixed  integer  algorithm. 
The  result  of  applying  the  algorithm  is  the  funding  level  of 
each  project  for  the  five  best  solutions  to  the  problem.  The 
decision  maker  is  not  simply  presented  with  the  optimal  solu¬ 
tion  for  two  reasons.  One  is  that  the  data  he  provides  and 
the  objective  function  that  is  derived  contain  errors  and  in¬ 
accuracies.  There  is  no  guarantee  then  that  the  "optimal" 
solution  is  really  optimal.  It  should  have  no  more  significance 
than  another  solution  whose  value  is  very  close.  These  ideas 
will  be  investigated  more  thoroughly  in  the  next  chapter. 

The  second  reason  is  that  there  may  be  other  considerations 
such  as  political  ones  which  could  not  be  incorporated  into 
the  utility  function  or  constraint  set.  If  the  decision  maker 
is  presented  with  a  range  of  solutions  with  near  "optimal" 
values,  then  he  may  choose  between  them  on  the  basis  of  such 
considerations  without  seriously  affecting  performance. 

The  project  selection  problem,  when  viewed  as  a 
multiple  objective  decision  problem,  can  be  solved  by  the  inter¬ 
active  procedure  described  above.  The  decision  maker  is  asked 
to  rate  sets  of  pseudo-projects  in  order  to  derive  a  measure 
of  the  utility  of  alternatives  as  a  function  of  their  criterion 
levels.  This  utility  function  is  combined  with  probability 
of  success  estimates  provided  by  the  decision  maker,  to  produce 
an  expression  for  the  expected  utility  of  any  research  and 
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development  program  which  is  used  as  the  objective  function 
in  a  mathematical  programming  algorithm.  When  this  objective 
function  is  combined  with  constraints  formulated  with  the  data 
provided  by  the  decision  maker,  the  problem  can  be  solved  with 
a  branch  and  bound  mixed  integer  algorithm.  The  usefulness  of 
the  resulting  solutions  and  some  of  the  ways  the  procedure 
may  be  extended  and  improved  are  discussed  in  the  next  chapter. 


Chapter  4 


THE  UTILITY  AND  ACCEPTABILITY  OF  THE  ALGORITHM 


Most  project  selection  algorithms  proposed  to  date 
have  received  little  acceptance  from  those  involved  in  the  fund¬ 
ing  of  research  and  development  projects.  Several  authors 
have  attempted  to  explain  this  phenomenon.  Baker  and  Pound  [5] 
suggest  three  reasons  for  it:  (1)  the  unrealistic  features 
and  assumptions  implicit  in  many  of  the  models,  (2)  the  onerous 
data  requirements  of  most  of  the  algorithms,  and  (3)  the  lack 
of  comprehensive  testing  and  reported  computational  experience 
with  real  problems. 

4.1  FEATURES  OF  THE  ALGORITHM 

The  algorithm  suggested  here  is  based  on  a  more  realistic 
model  of  the  project  selection  decision  situation.  In  particular 
profit  maximization  is  not  assumed  to  be  the  only  objective. 

For  many  organizations  such  as  government  fund  granting  agencies, 
this  objective  is  irrelevant.  In  other  cases  the  organization 
is  interested  in' profit  but  not  to  the’  excl us i on  of  other  goal 
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dimensions.  Often  once  a  reasonable  return  on  investment  is 
ensured,  other  criteria  are  triggered.  The  algorithm  suggested 
here  considers  all  criteria,  in  the  manner  and  to  the  extent 
they  are  reflected  in  the  decision  maker's  judgements.  This 
is  in  constrast  to  other  mul ti -cri teri a  project  selection 
algorithms,  such  as  scoring  type  models,  which  specify  pre¬ 
determined  methods  of  combining  goal  dimensions. 

Most  other  project  selection  algorithms  assume  the 
cost  of  a  project  is  fixed.  The  project  is  either  funded  at 
that  level  or  it  is  not  funded  at  all.  However,  the  cost  of 
any  project  is  only  an  estimate  of  the  amount  required  to  ensure 
a  reasonable  probabi 1 i ty  of  success.  Like  any  estimate  it  is 
subject  to  errors.  If  the  project  were  to  be  funded  at  an 
amount  less  than  its  "cost,"  then  it  would  be  less  likely  to 
succeed,  but  there  would  still  be  some  probability  of  success. 
Conversely,  if  the  project  were  funded  above  its  "cost,"  then 
it  would  be  more  likely  to  succeed.  This  is  true  only  within 
a  certain  range,  however,  For  any  project,  there  exists  a 
lower  funding  bound  such  that  if  it  is  allocated  any  less,  there 
is  virtually  no  probability  of  success,  and  an  upper  funding 
bound  such  that  if  it  is  allocated  any  more,  there  is  no  signifi¬ 
cant  increase  in  the  probability  it  will  succeed.  The  proposed 
algorithm  allows  the  amount  of  funding  to  fall  anywhere  between 
these  lower  and  upper  bounds.  The  relStionship  between  the 
funding  level  and  the  probability  of  success  of  the  project  is 


assumed  to  be  a  piecewise  linear  function  as  described  in  the 
last  chapter. 


These  two  features,  multiple-criteria  and  funding  in 
an  interval,  make  this  model  a  more  accurate  representation 
of  the  proj ect  selection  decision  situation  and  improve  the 
usefulness  of  the  algorithm  for  real  decision  situations. 


4.2  DATA  REQUIREMENTS 

The  data  required  by  the  algorithm  is  not  as  extensive 
as  that  required  by  most  other  algorithms.  The  decision  maker 
must  be  able  to  describe  each  project.  He  must  be  able  to 
specify  the  levels  of  all  relevant  criteria  that  each  project 
achieves,  and  the  amount  of  each  scarce  resource  each  project 
requires.  This  data  is  the  minimum  required  for  rational  selec¬ 
tion.  Often  project  descriptions  or  proposals  contain  all  the 
data  necessary.  However,  the  required  probability  of  success 
estimates  may  not  be  as  readily  available  and  often  must  be 
provided  by  the  proj  ect  evaluatior.  A  study  by  Souder  [74]  of 
the  validity  of  subjective  probability  of  success  estimates 
indicates  that  most  experienced  project  evaluators  can  assess 
the  probability  of  success  of  any  project  surprisingly  well, 
though  in  some  cases  this  assessment  is  not  accurately  communi¬ 
cated  for  political  reasons. 
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4.3  THE  USEFULNESS  OF  THE  ALGORITHM 

While  the  algorithm  was  not  tested  in  real  problem 
situations,^  it  has  some  attractive  features.  The  algorithm 
is  general  enough  to  be  useful  in  any  project  selection  decision 
situation.  Any  type  of  criteria  may  be  used  and  enough  data 
would  generally  be  available  on  even  somewhat  less-structured 
pure  research  proj ects.  It  is  probably  least  useful  however , 
for  deve 1 opment-type  projects  whose  only  goal  is  monetary.  Several 
of  the  other  algorithms  such  as  the  dynamic  programming  ones, 
are  better  suited  to  such  cases. 

Since  the  problem  is  a  combinational  one  (involves 
integer  constrained  variables),  the  value  of  the  optimal  sol¬ 
ution  is  not  a  continuous  function  of  the  value  of  the  right- 
hand-side  (resource  availabilities).  It  is  thus  possible 
that  a  slight  increase  in  the  available  amount  of  any  resource 
would  result  in  a  large  increase  in  the  value  of  the  optimal 
solution,  and  a  very  different  funding  pattern.  To  test  the 
sensitivity  of  the  solution  to  changes  in  the  right-hand-side, 
the  algorithm  may  be  re-run  without  re-deriving  the  utility 
functi on . 

Souder  has  discussed  the  "utility  and  acceptability" 
of  project  selection  algorithms.  He  suggests  (and  is  echoed 
by  Beattie  and  Reader  [6])  that  a  serious  shortcoming  of  the 

_  4 

A  sample  problem  and  its  solution  are  described  in 
the  Appendix. 


algorithms  is  that  only  one  solution,  the  supposedly  optimal 
solution,  is  provided.  This  defect  was  overcome  in  the  pro¬ 
posed  algorithm  by  providing  the  five  best  solutions  (if  five 
feasible  ones  exist).  Since  the  data  the  decision  maker  pro¬ 
vides  is  not  exact,  and  the  utility  function  derived  is  only 
an  approximate  measure  of  the  subjective  value  of  any  portfolio 
of  projects,  no  solution  can  be  considered  optimal.  Several 
solutions  may  have  almost  the  same  value,  and  given  the  accuracy 
of  the  data,  none  of  them  can  be  said  to  be  the  best.  There 
are  often  considerations  which  cannot  be  included  in  the  utility 
function  nor  in  the  problem's  constraints.  If  the  decision 
maker  has  a  set  of  solutions  to  choose  from,  he  can  discriminate 
among  them  on  the  basis  of  these  other  considerations.  In  that 
ay  the  algorithm  is  useful  as  a  tool  for  analyzing  the  decision 
situation.  It  simplifies  the  decision  by  reducing  the  number 
of  possible  choices.  However,  it  is  not  intended  to  replace 
the  decision  maker.  His  particular  skills  and  insights  are 
still  used.  He  is  in  fact  an  integral  part  of  the  algorithm. 

Rather  than  replacing  the  decision  maker,  the  algorithm 
attempts  to  formalize  some  of  his  thought  processes.  The 
utility  function  which  is  derived  is  a  formalization  of  pref¬ 
erence  choices  made  by  the  decision  maker.  It  can  be  applied 
to  other  decision  situations  and  would  result  in  consistent 
responses.  The  decision  maker  canno't  be  consistent.  He  is 

V. 

affected  by  extraneous  factors  he  can  neither  control  nor  com- 
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pensate  for.  Shepard  [70]  has  discussed  this  phenomenon  and 
terms  it  "subjective  non-optimality."  It  occurs  when  a  decision 
maker  choses  an  alternative  he  believes  to  be  best,  and  indeed, 
it  may  be  best  with  respect  to  his  current  state  of  mind  and 
the  stimuli  he  is  exposed  to.  However,  at  a  later  time, 
detached  from  these  stimuli  he  recognizes  that  the  choice  was 
not  optimal  with  respect  to  his  true  preferences.  The  door-to- 
door  encyclopedia  salesman  for  example,  and  other  "pressure" 
salesmen  capitalize  on  this  foible  of  human  nature.  Often,  a 
"paramorphic  representation"  of  a  decision  maker,  by  alleviating 
strong  situational  stimuli  can  out-perform  the  decision  maker 
by  reflecting  his  true  preferences.  Bowman  [10]  and  Kunreuther 
[53]  have  suggested  that  inconsistency  is  the  major  cause  of 
costly  decision  errors.  They  have  used  decision  models  based 
on  past  decisions  made  by  the  decision  maker  to  improve  his 
performance,  and  have  had  excellent  results  in  a  variety  of 
areas. 

Because  of  the  decision  maker’s  direct  involvement  in 

the  algorithm,  and  the  control  of  criteria  and  evaluation  methods 

allowed,  it  is  hoped  that  this  algorithm  will  be  more  acceptable 

as  a  decision  making  tool  than  previously  suggested  . ones .  It 

has  another  use  as  well,  however,  the  derived  utility  function 

can  be  used  for  diagnostic  purposes,  showing  the  decision  maker 

how  he  actually  makes  choices,  i.e.  vjhich  criteria  he  considers 

* 

most  important.  Any  discrepancies  between  these  criteria  and 
the  criteria  he  believes  he  uses  can  be  pointed  out.  Any 


illogical  features  can  thus  be  eliminated  from  his  utility 
function.  His  utility  function  can  be  compared  with  those  of 
other  decision  makers  in  the  organization.  Any  differences  can 
be  discussed,  and  possibly  an  organizational  objective  function 
can  be  agreed  upon.  A  better  utility  function  found  by  such 
analysis  can  be  used  to  help  analyze  future  decision  problems 
more  consistently.  The  optimization  section  of  the  algorithm 
can  be  used  alone,  if  a  utility  function  was  previously 
estimated. 

4.4  POSSIBLE  EXTENSIONS  AND  IMPROVEMENTS  TO  THE  ALGORITHM 

Several  extensions  and  improvements  to  the  algorithm 
are  possible.  The  piecewise  linear  function  can  be  extended  to 
include  more  than  two  pieces  in  the  hope  of  finding  a  better 
fitting  utility  function.  If  N  pieces  are  desired  N-l  poorly 
fitted  points  are  required  as  breakpoints.  The  current  algorithm 
may  be  easily  modified  to  find  these  break  points  and  the  co¬ 
efficients  a^  for  each  piece  i.  More  pseudo-project  ratings 
would  be  required  to  fit  a  line  accurately  in  each  of  the  new 
pieces.  The  piece-wise  linear  formulation  may  be  readily  ex¬ 
tended  to  include  N  pieces  by  introducing  s^,  i  =  1,  •••  ,  N-l. 
s’  is  the  amount  by  which  the  totallevel  of  criterion  k,  tk 

f*  h  * 

exceeds  the  i  breakpoints.  The  complete  formulation  is  as 
follows: 
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where  is  the  upper  bound  on  criterion  level  k  for  the  £ 

piece.  If  b[  ^  tk  <  b£+1  ,  then  s£  >  0  for  £  =  !,*••, f  and 

optimality  forces  s£  =  0  for  all  £  >  f,  since  (ak  -  ak+I)  <  0 

by  the  concavity  of  the  piecewise  linear  form.  The  value  of 

level  tfc  of  the  kth  criterion  is  then  ak  •  t^  minus  (ak  -  a£) 

for  each  unit  tk  exceeds  B*,  minus  (a'k  -  a ^ )  for  each  unit  tk 

exceeds  Bk,  etc.  The  amount  of  tk  that  falls  into  the  £th 

piece  is  thus  valued  at  ak.  This  formulation  can  be  readily 

solved  by  a  branch  and  bound  mixed  integer  algorithm  as  before. 

Another  possible  extension  would  be  to  approximate  some 

of  the  five  non-linear  utility  function  forms  with  a  piecewise 

linear  function.  There  may  be  cases  where  one  of  these  forms 

would  provide  a  better  fit  to  the  decision  maker's  utility  func*- 

tion  than  the  linear  form  or  a  directly  derived  piecewise  linear 

m  a . 

form.  The  conjunctive  form,  U  =  H  Xk  ,  is  concave  if  the  ak 

k*  1  m 

are  greater  than  1,  and  the  logarithmic  form,  U  =  £  ak  log  Xk, 

k=  1 

is  concave  if  the  ak  are  positive.  These  forms  could  be  accurately 
approximated  by  a  concave  piecewise  linearization.  The  piece- 
wise  linear  formulation  given  above  would  then  b*e  applicable. 

These  extensions  would  make  the  algorithm  more  flexible; 
able  to  model  more  decision  makers  and  decision  situations. 

In  the  great  majority  of  cases  however,  the  linear  form  or  the 
piecewise  linear  form  with  only  twp  pieces  will  be  as  accurate 

4 

n 

a  representation  of  the  decision  maker's  utility  function  as  the 
quality  of  the  data  •  j usti f i es . 


The  decision  model  developed  in  the  last  chapter 
is  applicable  to  projects  where  the  benefit  levels  are  related 
to  the  funding  level  through  the  probability  of  success  term. 

Such  projects  either  succeed,  in  which  case  all  the  benefits 
are  received,  or  fail,  in  which  case  none  of  the  benefits  are 
achieved.  Many  projects  however,  do  not  have  this  success- 
failure  structure.  The  benefit  levels  in  such  cases  are  directly 
related  to  funding,  with  no  probability  of  success  considerations 
necessary.  To  model  such  situations,  P  -  ( «. . )  or  a:  could  be 

J  J  J 

interpreted  as  the  proportion  of  the  benefit  level  which 
would  be  achieved  if  the  project  were  to  be  funded  at  i.  rather 

J 

than  the  requested  funding  level  f..  Similarly  P.(u.)  or 

J  J  J 

8 j  would  be  the  proportion  of  achieved  if  project  j  were 
to  be  funded  at  P  4 ( x  - )  would  then  be  linear  between  u- 

J  J  J  J 

and  j . 

W  '  Vj  +  (6i  -  “j>  yy 

where  x.  =  i.z.  +  (8.  -  a.  )  y. .  Often  projects  are  judged 

J  J  J  J  J  J 

on  sets  of  criteria  which  fall  into  both  categories.  For 

example,  consider  a  project  with  the  aim  of  developing  a  new 

optical  instrument.  If  two  of  the  criteria  used  to  evaluate 

it  are  export  sales,  and  jobs  created,  then  the  former  falls 

into  the  success-failure  category,  vJh i\l e  the  latter  depends 

* 

directly  on  funding  level  and  is  essentially  independent  of 
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.... 


the  outcome  of  the  project.  These  situations  may  be  modelled 
by  considering  Pj(Xj)  t0  I36  a  probability  in  the  one  case  and 
a  proportion  in  the  other. 

Other  types  of  projects  that  the  algorithm  may  be 
modified  to  handle  are  those  which  extend  across  more  than 
one  planning  period.  For  the  linear  utility  function  the 
formulation  would  be  as  follows: 


max  £  £  a.  £  [a,.z..  +  (3..  -  <*  )]  X... 

j  s]  |(s]  ^  j =  1  ‘J'J  ^  J  1  J 


1 1 

[1uz(j  ♦  (“(j  -  «  Bt  v  1  <" 


n 


I  !ui  -  t»,j  -  »u>  y(j]  <  ru  (2) 

J  i  j 


yij  4  Zij 


V  i  ,j 


other  constraints 


-  0  or  1.  0  «  y,j  <  1 


V  t.j 


where  the  index  i  *  1,...  P  refers  to  the  planning  period, 

4 

and  all  the  variables  are  as  previously  defined. 

The  project  selection  decision  model  presented  here 


attempts  to  be  a  more  realistic  representation  of  the  real 
decision  situation.  Because  of  this  greater  realism  and  the 
participation  required  of  the  decision  maker  it  is  hoped  that 
it  will  be  a  more  acceptable  and  useful  tool  for  analyzing 
project  selection  decision  situations  than  previously  developed 


algorithms . 
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APPENDIX  -  A  SAMPLE  PROBLEM 


Consider  a  government  fund-granting  agency  which 
has  a  number  of  projects  under  consideration.  In  order  to 
use  the  project  selection  algorithm  as  a  decision  tool  in 
analyzing  the  situation,  three  types  of  inputs  to  the  program 
are  required. 

The  first  input  required  is  i nf orma ti on  about  goals 
and  constraints.  The  decision  maker,  in  consultation  with 
the  policy  analysis  staff  of  the  agency  must  decide  upon  the 
goals  which  should  be  achieved  by  the  particular  program, 
and  the  indicators  that  best  reflect  achievement  with  respect 
to  these  goals.  Often  goals  may  be  cast  as  constraints. 

For  example,  one  goal  might  be  the  encouragement  of  economic 
growth  in  depressed  regions.  This  goal  could  be  represented 
by  a  requirement  that  a  minimal  proportion  of  the  funds  be 
spent  in  such  regions.  Other  goals  such  as  decreasing  unem¬ 
ployment,  increasing  the  GNP,  and  improving  the  quality  of 
life  could  be  included  directly  in  the  objective  function. 

The  resources  required  by  the  projects  must  then  be  appraised 
in  terms  of  total  resources  available.  Clearly  capital,  or 
funds  to  be  allocated,  is  considered  to  be  a  scarce  resource. 
Other  examples  of  scarce  resources  may  be  various  types  of 
skilled  labour,  energy,  or  specific  types  of  equipment.  Re- 

4 

source  availabilities  may  Impose  constraints  on  subsets  of 
projects  or  on  all  of  them. 
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The  second  type  of  input  necessary  for  the  program 
is  information  from  the  project  evaluator.  The  evaluator  is 
responsible  for  estimating  the  probability  of  success  of  each 
project.  Consequently,  he  must  evaluate  the  competence  and 
capabilities  of  the  applicants,  the  merits  of  the  proposal  in 
terms  of  technological  requirements,  and  finally,  the  compat- 
ability  of  the  applicants  capabilities  and  the  proposed  project. 
On  the  basis  of  these  considerations,  he  should  be  able  to 
indicate  the  lower  funding  bound  or  critical  point  below  which 
the  project  would  have  little  probability  of  success  and  the 
upper  funding  bound  or  satiation  point  above  which  an  increase 
in  funding  would  not  be  reflected  In  a  significant  increase 
in  the  probability  of  success  of  the  project.  Probability 
of  success  estimates  given  that  the  project  is  funded  at  the 
two  bounds  are  required. 

The  applicant  must  provide  the  third  phase  of  input 
to  the  program.  He  must  specify  the  level  of  funding  desired 
for  the  project,  and  describe  the  project  in  terms  of  resources 
required,  benefits  produced,  and  other  costs  incurred.^  A 
proposed  schedule  is  also  necessary  so  that  the  decision  maker 
can  discount  the  benefits  to  facilitate  comparisons  between 
projects  with  different  timing  patterns. 

^"Other  costs"  may  include  social  costs  and  exter¬ 
nalities  such  as  an  increase  in  pollution  level. 

* 

* 


* 
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In  the  example  chosen,  the  government  agency  has  the 
following  three  goals;  1)  increasing  employment  2)  improving  the 
balance  of  payments,  and  3)  improving  the  quality  of  life.  Achieve¬ 
ment  with  respect  to  the  first  goal  can  be  measured  directly  by 
the  number  of  jobs  created  by  each  projects.  Two  indicators  may 
be  used  to  measure  achievement  with  respect  to  the  second  goal; 

1)  export  sales,  and  2)  added  value.  Quality  of  life  according  to 
this  agency,  concerns  stabilization  rathern  than  improvement,  and 
consequently  the  pollution  produced  by  each  project  is  considered 
to  be  an  indicator  related  to  the  third  goal.  The  agency  has  five 
projects  under  consideration  with  the  following  characteristics. 


Project 

1 

mumn 

Project 

4 

Project 

5 

Resource 

capital  (million  $) 

1 

5 

•  15 

2 

n 

Required 

skilled  labour  1 

300 

100 

- 

skilled  labour  2 

- 

100 

50 

wm 

Benefits 

jobs  created 

150 

200 

Produced 

export  sales 
($10,000) 

2 

3 

5 

2 

6 

value  added 
($10,000) 

5 

10 

30 

5 

10 

pollution  created 

-1 

-1 

-10 

-1 

-13 

Funding  (in 

lower  bound 

1 

2 

10 

1 

5 

million  $) 

requested  level 

1 

5 

15 

10 

upper  bound 

2 

15 

20 

15 

at  lower  bound 

.8 

.5 

.7 

.8 

.7 

at  upper  bound 

r 

1.0 

.85 

.98 

1.0 

.95 

TABLE  II  * 

SAMPLE  PROBLEM  DATA 

The  following  pages  are  the  output  from  the  interactive  project 
selection  algorithm  applied  to  this  problem. 
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WfcLCOHE  TO  'MODEM',  A  MULTIPLE  OBJECTIVE  DECISION  MAKING  PROGRAM. 
MODEM  HAY  HELP  YOU  DECIDE  WHICH  OF  A  NUMBER  OF  PROJECTS  TO  SUPPORT  AMD 
TO  WHAT  EXTENT  SUPPORT  SHOULD  BE  GIVEN. 

BACKUP:  ENTERING  *  BACK  *  WILL  CAUSE  REVERSION  TO  A  PREVIOUS  QUESTION. 

TERMINATION:  ENTERING  'END',  ‘HALT*  OR  'STOP*  WILL  STOP  MODEM  IMMEDIATELY. 

UOW  MANY  PROJECTS  ARE  THERE?  (MAXIMUM  IS  10.) 

5 

WHAT  IS  THE  NAME  OF  PROJECT  1?  one 

WHAT  IS  THE  NAME  OF  PROJECT  2?  two 

WHAT  IS  THE  NAME  OF  PROJECT  3?  three 

WHAT  IS  THE  NAME  OF  PROJECT  4?  f0Ur 

WHAT  IS  THE  NAME  OF  PROJECT  5?  five 

WHAT  IS  THE  TOTAL  NUMBER  OF  BENEFITS  (OBJECTIVES,  PAYOFFS,  OUTPUTS) 

FOR  ALL  OF  THESE  PROJECTS?  (MAXIMUM  OF  10.) 

4 

WHAT  IS  THE  NAME  OF  BENEFIT  1?  jobs 

WHAT  IS  THE  NAME  OF  BENEFIT  2?  export  sales 

what  is  the  name  of  benefit  3?  value  added 

WHAT  IS  THE  NAME  OF  BENEFIT  4?  pollution 

TIME  TO  FILL  IN  THE  PROJECT-BENEFIT  MATRIX. 

WHAT  IS  THE  JOBS  PAYOFF  FOR  ONE?  500 
WHAT  IS  THE  EXPORT  SALES  PAYOFF  FOR  ONE?  2 
WHAT  IS  THE  VALUE  ADDED  PAYOFF  FOR  ONE?  5 
WHAT  IS  THE  POLLUTION  PAYOFF  FOR  ONE?  -) 

WHAT  IS  THE  JOBS  PAYOFF  FOR  TWO?  ) 00 
WHAT  IS  THE  EXPORT  SALES  PAYOFF  FOR  TWO?  3 
WHAT  IS  THE  VALUE  ADDED  PAYOFF  FOR  TWO?  )0 
WHAT  IS  THE  POLLUTION  PAYOFF  FOR  TWO?  -5 
WHAT  IS  THE  JOBS  PAYOFF  FOR  THREE?  150 

1 

WHAT  IS  THE  EXPORT  SALES  PAYOFF  FOR  THREE?  5 
WHAT  IS  THE  VALUE  ADDED  PAYOFF  FOR  THREE?  30 
WHAT  IS  THE  POLLUTION  PAYOFF  FOR  THREE?  -10 


WHAT  IS  THE  JOBS  PAYOFF  FOR  FOUR?  300 

WHAT  IS  THE  EXPORT  SALES  PAYOFF  FOR  POUR?  2 

WHAT  IS  THE  VALUE  ADDED  PAYOFF  FOR  FOUR?  5 

WHAT  IS  THE  POLLUTION  PAYOPF  FOR  FOUR?  -1 

WHAT  IS  THE  JOBS  PAYOFF  FOR  PIVE?  200 

WHAT  IS  THE  EXPORT  SALES  PAYOFF  FOR  PIVE?  6 

WHAT  IS  THE  VALUE  ADDED  PAYOFF  FOR  FIVE?  10 

WHAT  IS  THE  POLLUTION  PAYOFF  FOR  FIVE?  -13 

WHAT  IS  THE  INITIAL  REQUESTED  LEVEL  OF  FUNDING  FOR  ONE?  1 

WHAT  IS  THE  MINIMUM  ACCEPTABLE  SUPPORT  FOR  ONE?  1 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  ONE  AT  THIS  FUNDING? 

ANSWER  SHOULD  BE  IN  INTEGRAL  PERCENT  —  A  NUMBER  BETWEEN  0  AND  100. 

80 

WHAT  IS  THE  MAXIMUM  ACCEPTABLE  SUPPORT  FOR  ONE?  2 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  ONE  AT  THIS  FUNDING?  100 

WHAT  IS  THE  INITIAL  REQUESTED  LEVEL  OF  FUNDING  FOR  TWO? 5 

WHAT  IS  THE  MINIMUM  ACCEPTABLE  SUPPORT  FOR  TWO?  2 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  TWO  AT  THIS  FUNDING?  50 

WHAT  IS  THE  MAXIMUM  ACCEPTABLE  SUPPORT  FOR  TWO?  15 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  TWO  AT  THIS  FUNDING?  85 

WHAT  IS  THE  INITIAL  REQUESTED  LEVEL  OF  FUNDING  FOR  THREE?  15 

WHAT  IS  THE  MINIMUM  ACCEPTABLE  SUPPORT  FOR  THREE?  10 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  THREE  AT  THIS  FUNDING?  70 

WHAT  IS  THE  MAXIMUM  ACCEPTABLE  SUPPORT  FOR  THREE?  20 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  THREE  AT  THIS  PUNDING?  98 

WHAT  IS  THE  INITIAL  REQUESTED  LEVEL  OF  FUNDING  FOR  FOUR?  2 

WHAT  IS  THE  MINIMUM  ACCEPTABLE  SUPPORT  FOR  FOUR?  1 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  FOUR  AT  THIS  FUNDING?  80 

WHAT  IS  THE  MAXIMUM  ACCEPTABLE  SUPPORT  FOR  FOUR?  4 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  FOUR  AT  THIS  FUNDING?  100 


WHAT  IS  THE  INITIAL  REQUESTED  LEVEL  OF  FUNDING  FOR  PIVE?  10 
WHAT  IS  THE  MINIMUM  ACCEPTABLE  SUPPORT  FOR  FIVE?  5 


97 


WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  FIVE  AT  THIS  FUNDING?  70 

WHAT  IS  THE  MAXIMUM  ACCEPTABLE  SUPPORT  FOR  FIVE?  15 

WHAT  IS  THE  PROBABILITY  OF  SUCCESS  OF  FIVE  AT  THIS  FUNDING?  95 

HOW  MANY  DISTINCT  RESOURCES  ARE  THERE  FOR  THE  5  PROJECTS 
(MAXIMUM  OF  10.) 

3 

WHAT  IS  THE  NAME  OF  RESOURCE  1?  capital 

WHAT  IS  THE  NAME  OF  RESOURCE  2?  labor  1 

WHAT  IS  THE  NAME  OF  RESOURCE  3?  labor  2 


PLEAS 

IE  FILL 

IN 

THE  PROJECT-RESOURCES  MATRIX. 

WHAT 

AMOUNT 

OF 

CAPITAL 

IS 

REQUIRED 

FOR 

ONE?  1 

WHAT 

AMOUNT 

OF 

LABOR  1 

IS 

REQUIRED 

FOR 

ONE?  300 

WHAT 

AMOUNT 

OF 

LABOR  2 

IS 

REQUIRED 

FOR 

ONE? 

0  . 

WHAT 

AMOUNT 

OF 

CAPITAL 

IS 

REQUIRED 

FOR 

TWO? 

5 

WHAT 

AMOUNT 

OF 

LABOR  1 

IS 

REQUIRED 

FOR 

TWO?  100 

WHAT 

AMOUNT 

OF 

LABOR  2 

IS 

REQUIRED 

FOR 

TWO? 

0 

WHAT 

AMOUNT 

OF 

CAPITAL 

IS 

REQUIRED 

FOR 

THREE?  15 

WHAT 

AMOUNT 

OF 

LABOR  1 

IS 

REQUIRED 

FOR 

THREE? 

0 

WHAT 

AMOUNT 

OF 

LABOR  2 

IS 

REQUIRED 

FOR 

THBEE? 

100 

WHAT 

AMOUNT 

OF 

CAPITAL 

IS 

REQUIRED 

FOR 

FOUR? 

2 

WHAT 

AMOUNT 

OF 

LABOR  1 

IS 

REQUIRED 

FOR 

FOUR? 

0 

WHAT 

AMOUNT 

OF 

LABOR  2 

IS 

REQUIRED 

FOR 

FOUR? 

50 

WHAT 

AMOUNT 

OF 

CAPITAL 

IS 

REQUIRED 

FOR 

FIVE? 

100 

WHAT 

AMOUNT 

OF 

LABOR  1 

IS 

REQUIRED 

FOR 

FIVE? 

150 

WHAT 

AMOUNT 

OF 

LABOR  2 

IS 

REQUIRED 

FOR 

FIVE? 

* 

0 

WHAT 

IS  THE 

TOTAL  MAXIMUM 

AVAILABLE 

!  FOR  CAPITAL?  20 

WHAT 

IS  THE 

TOTAL  MAXIMUM 

AVAILABLE  FOR  LABOR 

1?  450 

WHAT 

IS  THE 

TOTAL  MAXIMUM 

AVAILABLE 

!  FOR  LABOR 

2?  120 

MOULD  YOU  LIKE  TO  DERIVE  A  UTILITY  FUNCTION?  yes 

THE  POLLOHING  ABF  PSEUDO-PROJECT  PROFILES 
WITH  BENEFITS  ORDERED  THE  MAY  YOU  PRESENTED  THEM. 

PLEASE  SCORE  EACH  PSEUDO-PROJECT  PROM  1  (HORST  POSSIBLE) 
CORRESPONDING  TO  A  BENEFIT  PROPILE  OF 

40  1  2  -19 

TO  100  (BEST  POSSIBLE)  CORRESPONDING  TO  A  BENEFIT  PROFILE  OF 
1000  9  40  0 

BENEFITS: 

12  3  4 

PSEUDO-PROJECT  1: 

716  2  5  -10 

SCORE?  39 
PSEUDO-PROJECT  2: 

507  4  22  -4 

SCORE?  59 
PSEUDO-PROJECT  3: 

873  7  26  -14 

SCORE?  65 
PSEUDO-PROJECT  4: 

146  7  34  -2 

SCORE?  60 
PSEUDO-PROJECT  5: 

53  8  13  0 

SCORE?  51 
PSEUDO-PROJECT  6: 

264  6  30  -  16 

SCORE?  35 
PSEUDO-PROJECT  7: 

895  3  39  -3 

SCORE?  76 
PSEUDO-PROJECT  8: 

808  7  10  -5 

SCORE?  72 
PSEUDO-PROJECT  9: 

644  7  19  -15 

SCORE?  51 
PSEUDO-PROJECT  10; 

309  7  35  -4 

SCORE?  67 
PSEUDO-PROJECT  11: 

851  4  17  -12 
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SCORE?  57 
PSEUDO-PROJECT  12: 

422  4  39 

SCORE?  42 
PSEUDO-PROJECT  13: 

988  5  35 

SCORE?  65 
PSEUDO-PROJECT  14: 

140  4  9 

SCORE?  30 


-9 


-10 


PSEUDO-PPOJECT  15: 

948  2  39 

SCORE?  65 
PSEUDO -PROJECT  16: 

816  8  6  -8 

SCORE?  66 


FOLLOWING  IS  AN  ORDERED  DISPLAY  OF  THE  PS EU DO- PROJ ECTS 


PROJ  # 

SCORE 

BENEFITS 

1 

76 

895 

3 

39 

-3 

2 

72 

808 

7 

10 

-5 

3 

67 

309 

7 

35 

-4 

4 

66 

816 

8 

6 

-8 

5 

65 

873 

7 

26 

-14 

6 

65 

988 

5 

35 

-15 

7 

65 

948 

2 

39 

-10 

8 

60 

146 

7 

34 

-2 

9 

59 

507 

4 

22 

-4 

10 

57 

851 

4 

17 

-12 

11 

51 

53 

8 

13 

0 

12 

51 

644 

7 

19 

-15 

13 

42 

422 

4 

39 

-16 

14 

39 

716 

2 

5 

-10 

15 

35 

264 

6 

30 

-16 

16 

30 

140 

4 

9 

-9 

YOU  HAY  NOW  CHANGE  SCORES. 

WHICH  PSEUDO-PROJECT’S 

SCORE 

DC 

YOU 

WISH  TO  CHANGE? 

A  ZERO 

MEANS 

/i 

THAT  NO 

WORE 

CHANGES  ARE  DESIRED. 

WHAT  IS  4 • S  NEW 

SCORE? 

64 

FOLLOW ING 

IS  AN 

ORDERED 

DISPLAY 

OF 

THE  PSEUDO-PROJECTS 

PROJ  # 

SCORE 

BENEFITS 

1 

76 

895 

3 

39 

-3 

2 

72 

808 

7 

10 

-5 

3 

67 

309 

7 

35 

-4 

4 

65 

873 

7 

26 

-14 

5 

65 

988 

5 

35 

-15 

6 

65 

948 

2 

39 

-10 

7 

64 

816 

8 

6 

-8 

8 

60 

146 

7 

34 

-2 

9 

59 

507 

4 

22 

-4 

10 

57 

851 

4 

17 

-12 

1 1 

51 

53 

8 

13 

0 

12 

51 

644 

7 

19 

-15 

13 

42 

422 

4 

39 

-16 

14 

39 

716 

2 

5 

-10 

15 

35 

264 

6 

30 

-16 

16 

30 

140 

4 

9 

-9  . 

YOU  HAY  NOW  CHANGE  SCORES.  * 

WHICH  PSEUDO-PROJECT’S  SCORE  DO  YOU  WISH  TO  CHANGE? 

A  ZERO  BEANS  THAT  NO  HORE  CHANGES  ARE  DESIRED.  0+ 


FINAL 

RANKING  OF 

PSEUDO- 

PROJECTS 

>: 

PROJ 

#  SCORE 

BENEFITS 

1 

76 

895 

3 

39 

2 

72 

808 

7 

10 

3 

67 

309 

7 

35 

4 

65 

873 

7 

26 

5 

65 

988 

5 

35 

6 

65 

948 

2 

39 

7 

64 

816 

8 

6 

8 

60 

146 

7 

34 

9 

59 

507 

4 

22 

10 

57 

851 

4 

17 

1 1 

51 

53 

8 

13 

12 

51 

644 

7 

19 

13 

42 

422 

4 

39 

14 

39 

716 

2 

5 

15 

35 

264 

6 

30 

16 

30 

140 

4 

9 

COEFFS:  4.6E- 

02  4.9E 

00  6.5E-01 

PSEUDO-PROJECT 

ACTUAL 

PREDICTED 

1 

76 

76.64 

2 

72 

70.38 

3 

67 

65.20 

4 

65 

70.82 

5 

65 

70.78 

6 

65 

64.  14 

7 

64 

68.70 

8 

60 

59.95 

9 

59 

51.23 

10 

57 

52.25 

11 

51 

49.78 

12 

51 

54.34 

13 

42 

41.15 

14 

39 

31.42 

15 

35 

37.78 

16 

30 

18.78 

SCORING  BY  MISTAKES* DISTANCE  METHOD 
SCORE  »  4 

R-SQUARE  ■  383.8 

CORRELATION  BY  PEARSON’S  HETHOC: 
fl  =  0.96 


SPEARMAN  CORRELATIONS: 

R-SOB-S  *  0.94 


DISTANCE* ERR OR  METHOD: 

TAO  =  0.82 
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LEXICOGRAPHIC  ORDERING  OF  BENEFITS: 
BENEFIT  NUMBER  CF  ORDER  ERRORS 
1  6 

2  4 

3  7 

4  7 


IS  THIS  UTILITY  FUNCTION  ACCURATE  ENOUGH?  no 

HORST  POINT  IS  16 

ANOTHER  PSEUDO-PROJECT  TO  SCORE: 

136  5  9  -9 

SCORE?  36 


ILLOWING 
PROJ  # 

IS  AN 
SCORE 

ORDERED  DISPLAY 
BENEFITS 

OF 

THE  P! 

1 

76 

895 

3 

39 

-3 

2 

72 

808 

7 

10 

-5 

3 

67 

309 

7 

35 

-4 

4 

65 

873 

7 

26 

-14 

5 

65 

988 

5 

35 

-15 

6 

65 

948 

2 

39 

-10 

7 

64 

816 

8 

6 

-8 

8 

60 

146 

7 

34 

-2 

9 

59 

507 

4 

22 

-4 

10 

57 

851 

4 

17 

-12 

11 

51 

53 

8 

13 

0 

12 

51 

644 

7 

19 

-15 

13 

42 

422 

4  . 

39 

-16 

14 

39 

716 

2 

5 

-10 

15 

36 

136 

5 

9 

-9 

16 

35 

264 

6 

30 

-16 

CHANGE  SCORES. 


IICH 

PSEUDO-PROJECT'S 

SCORE 

DO  YOU 

WISH 

A  ZERO 

MEANS 

THAT  NO 

MORE 

CHANGES 

ARE 

NAL 

RANKING  OF 

u 

PSEUDO 

-PROJECTS: 

PROJ 

* 

SCORE 

BENEFITS 

1 

76 

895 

3 

39 

-3 

2 

72 

808 

7 

10 

-5 

3 

67 

309 

7 

35 

-4 

4 

65 

873 

7 

26 

-14 

5 

65 

988 

5 

35 

-15 

6 

65 

948 

2 

39 

-10 

7 

64 

816 

a 

6 

-8 

8 

60 

146 

7 

34 

-2 

9 

59 

507 

4 

22 

-4 

10 

57 

851 

4 

17 

-12 

11 

51 

53 

8 

13 

0 

12 

51 

644 

7 

19 

-15 

13 

42 

422 

4 

39 

-16 

14 

39 

716 

2 

5 

-10 

TO  CHANGE? 
DESIRED. 
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15  36  136 

16  35  264 


COBFFS:  4 .52-02  5.0E 


-PROJECT 

ACTUAL 

1 

76 

2 

72 

3 

67 

4 

65 

5 

65 

6 

65 

7 

64 

8 

60 

9 

59 

10 

57 

11 

51 

12 

51 

13 

42 

14 

39 

15 

36 

16 

35 

5  9  -9 

6  30  -16 


00  6.4E-01  1.4E  00 


PREDICTED 

76.22 

70.65 
65.44 
70.99 
70.64 

63.66 
69.  12 
60.25 

51.23 
52.22 
50.39 
54.68 
41.11 
31.34 
23.86 
38.08 


SCORING  BY  MISTAKES*DISTANCE  METHOD: 
SCORE  =  7 

H-SQUARE  =  413.5 


CORRELATION  BY  PEARSON'S  METHOD: 
R  =  0.95 


SPEARMAN  CORRELATIONS: 

R-SUB-S  =  0.94 


DISTANCE* ERROR  METHOD: 

TAU  =  0.82 


LEXICOGRAPHIC  ORDERING  OF  BENEFITS: 
BENEFIT  NUMBER  CF  ORDER  ERRORS 

1  7 

2  5 

3  8 

4  7 


IS  THIS  UTILITY  FUNCTION  ACCURATE  ENOUGH?  no 
WORST  POINT  IS  15 

4 

THE  FOLLOWING  ARE  PSEUDO-PROJECT  PROFILES.  ’ 

WITH  BENEFITS  ORDERED  THE  WAY  YOU  PRESENTED  THEM. 

PLEASE  SCORE  EACH  PSEUDO-PROJECT  FROM  1  (WORST  POSSIBLE) 
CORRESPONDING  TO  A  BENEFIT  PROFILE  OF 
40  1  2-19 


MMUUi 


TO  100  (BEST  POSSIBLE)  CORRESPONDING  TO  A  BENEFIT  PROFILE  OF 
1000  9  40  0 

BENEFITS: 


1  2 

3 

4 

PSEUDO-PROJECT 

1: 

895  3 

39 

-3 

SCORE*  76 

PSEUDO-PROJECT 

2: 

808  7 

10 

-5 

SCORE®  72 

PSEUDO -PROJECT 

3: 

309  7 

35 

-4 

SCORE®  67 

PSEUDO-PROJECT 

4: 

873  7 

26 

-14 

SCORE®  65 

PS EU DO-PROJECT 

5: 

988  5 

35 

-15 

SCORE®  65 

PSEUDO-PROJECT 

6 : 

948  2 

39 

-10 

SCORE®  65 

PS  EU  DO -PROJECT 

7: 

816  8 

6 

-8 

SCORE®  64 

PSEUDO-PROJECT 

8: 

146  7 

34 

-2 

SCORE®  60 

PS  EU DO-PROJECT 

9: 

507  4 

22 

-4 

SCORE®  59 

PSEUDO-PROJECT 

10: 

851  4 

17 

-12 

SCORE®  57 

PSEU  CO-PROJECT 

11: 

53  8 

13 

0 

SCORE®  51 

PSEUDO-PROJECT 

12: 

644  7 

19 

-  15 

SCORE®  51 

PS EU DO-PROJECT 

13: 

422  4 

39 

-16 

SCORE®  42 

PSEUDO-PROJECT 

14: 

716  2 

5 

-10 

SCORE®  39 

PS EU DO-PROJECT 

15: 

136  5 

9 

-9 

SCORE*  36 

PSEUDO-PROJECT 

16: 

264  6 

30 

-16 

SCORE®  35 

PS  EU  DO -PROJECT 

1: 

860  7 

27 

-12 

SCORE?  70 
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PSEO DO-PROJECT 

2: 

421  9 

13 

-11 

SCORE?  59 

PSEUDO-PROJECT 

3: 

272  8 

38 

-  12 

SCORE?  58 

PS  EO CO-PROJECT 

4: 

772  5 

15 

-9 

SCORE?  62 

PSEUDO-PROJECT 

5: 

624  8 

11 

-13 

SCORE?  54 

PSEUDO-PROJECT 

6: 

653  4 

30 

-17 

SCORE?  41 

PSEUDO-PROJECT 

7: 

477  3 

15 

-  10 

SCORE?  42 

PS EU CO-PROJECT 

8: 

877  2 

19 

-3 

SCORE?  60 

PSEUDO-PROJECT 

9: 

635  6 

5 

-19 

SCORE?  21 

PS EU DO-PROJ ECT 

10: 

618  7 

28 

-11 

SCORE?  67 

PSEUDO-PROJECT 

11: 

48  6 

4 

-17 

SCORE?  1 

PS  EU  DO -PROJ  ECT 

12: 

631  5 

17 

-14 

SCORE?  71 

PSEUDO-PROJECT 

13: 

584  3 

31 

-7 

SCORE?  57 

PS EU DO-PROJ ECT 

14: 

955  8 

34 

-7 

SCORE?  90 

PSEUDO-PROJECT 

15: 

531  5 

33 

-10 

SCORE?  62 

PSEUDO-PROJECT 

16: 

760  2 

21 

-16 

SCORE?  31 


FOLLOWING  IS  AN  ORDERED  DISPLAY  OF  THE  PS EU DO-PROJ ECTS : 
PROJ  »  SCORE  BENEFITS 


1 

90 

955 

8 

34 

-7 

2 

71 

631 

5 

17 

-14  . 

3 

70 

860 

7 

27 

-12  ‘ 

4 

67 

618 

7 

28 

-11  * 

5 

62 

772 

5 

15 

-9 

6 

62 

531 

5 

33 

-10 

7 

60 

877 

2 

19 

-3 

•*-.  --v  v 

X?  "-  '.a 

'  - 

~  - 

WWP.’JiL-L .  "V"  - - - — 

- 

105 

8 

59 

421 

9 

13 

-11 

:i 

9 

58 

272 

8 

38 

-12 

10 

57 

584 

3 

31 

-7 

11 

54 

624 

8 

11 

-13 

12 

42 

477 

3 

15 

-10 

13 

41 

653 

4 

30 

-17 

14 

31 

760 

2 

21 

-16 

^  15 

21 

635 

6 

5 

-  19 

; *  16 

1 

48 

6 

4 

-17 

I  YOU  HAY  NOW  CHANGE  SCORES. 

I  ;  which  pseudo- project* s 

SCORE 

DO  YOU 

WISH 

TO  CHANGE? 

ft  A  ZERO 

BEANS 

THAT  NO 

BORE 

CHANGES 

ARE 

DESIRED. 

1  ’ 

0 

I  AN  ORDINA1 

DOMINANCE  CONFLICT 

IN  SCORING 

HAS  BEEN  FOUND  BETWEEN  2  AND  3. 

I  THESE  TWO 

ARE  INDICATED 

BY  ASTERISKS 

IN  THE  FOLLOWING  PRESENTATION. 

ft  PLEASE  CHANGE  ONE  CR  BOTH  SCORES  SO 

THAT 

A  CONFLICT  WILL  NOT  OCCUR. 

PROJ  > 

SCORE 

BENEFITS 

1 

90 

955 

8 

34 

-7 

*  2 

71 

631 

5 

17 

-14 

*  3 

70 

860 

7 

27 

-12 

;  f  4 

67 

618 

7 

28 

-11 

:  l  5 

62 

772 

5 

15 

-9 

[  6 

62 

531 

5 

33 

-10 

!■  7 

60 

877 

2 

19 

-3 

8 

59 

421 

9 

13 

-11 

- 

9 

58 

272 

8 

38 

-12 

10 

57 

584 

3 

31 

-7 

11 

54 

624 

8 

1 1 

-13 

12 

42 

477 

3 

15 

-10 

-  13 

41 

653 

4 

30 

-17 

,  1“ 

31 

760 

2 

21 

-16 

[  15 

21 

635 

6 

5 

-19 

,6 

1 

48 

6 

4 

-17 

|.|  WHICH  PSEUDO-PROJECT’S 

SCORE 

DO  YOU 

WISH 

TO  CHANGE 

A  ZERO 

BEANS 

THAT  NO 
2 

BORE 

CHANGES 

ARE 

DESIRED. 

WHAT  IS  2’ 

S  NEW 

SCORE? 

47 

FOLLOWING 

IS  AN 

ORDERED 

DISPLAY  OF  THE  PSEUDO-PROJECTS: 

PROJ  i 

SCORE 

BENEFITS 

1 

90 

955 

8 

34 

-7 

2 

70 

860 

7 

27 

-12 

3 

67 

618 

7 

28 

-11 

4 

62 

772 

5 

15 

-9 

5 

62 

531 

5 

33 

-10 

6 

60 

877 

2 

19 

-3 

7 

59 

421 

9 

13 

-11 

< 

8 

58 

272 

8 

38 

-12 

4 

\  j 

9 

57 

584 

3 

31 

-7 

n 

\  i 

10 

54 

624 

8 

1 1 

-13 

11 

47 

631 

5 

17 

-14 

i  12 

42 

477 

3 

15 

-10 

41 

653 

4 

30 

-17 

1 

i  ‘ 

31 

760 

2 

21 

-16 

. .  _  _ _  _ 
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15 

21 

6  35 

6 

5 

-19 

16 

1 

48 

6 

4 

-17 

YOU  MAY  NOW  CHANGE  SCORES. 

WHICH  PSEUDO-PROJECT'S 

SCORE  CO 

YOU 

WISH  TO 

CHANGE? 

A  ZERO 

MEANS 

THAT  NO 
n 

MORE  CHANGES  ARE  DESIRED. 

FINAL  RANKING  OF  PSEUDO 

-PROJECTS 

;  ; 

PROJ  * 

SCORE 

BENEFITS 

1 

90 

955 

8 

34 

-7 

2 

70 

860 

7 

27 

-12 

3 

67 

618 

7 

28 

-11 

4 

62 

772 

5 

15 

-9 

5 

62 

531 

5 

33 

-10 

6 

60 

877 

2 

19 

-3 

7 

59 

421 

9 

13 

-11 

8 

58 

272 

8 

38 

-12 

9 

57 

584 

3 

31 

-7 

10 

54 

624 

8 

1 1 

-13 

11 

47 

631 

5 

17 

-14 

12 

42 

477 

3 

15 

-10 

13 

41 

653 

4 

30 

-17 

14 

31 

760 

2 

21 

-16 

15 

21 

635 

6 

5 

-19 

16 

1 

48 

6 

4 

-17 

BENEFIT 

1,  COEFFS:  0 

.0519567 

0. 

0300714, 

BREAKPOINT: 

BENEFIT 

2,  COEFFS:  4 

.6440487 

3. 

3582411, 

BREAKPOINT: 

BENEFIT 

3,  COEFFS:  0 

.6867285 

0. 

5346403, 

BREAKPOINT: 

BENEFIT 

4,  COEFFS:  -0 

.8256531 

2. 

1860657, 

BREAKPOINT: 

PSEUDO-PROJECT 

ACTUAL 

PREDICTED 

1 

76 

78.62 

2 

72 

72.03 

3 

67 

70.07 

4 

65 

64.69 

5 

65 

64.06 

6 

65 

63.27 

7 

64 

66.32 

8 

60 

59.4  1 

9 

59 

63.33 

10 

57 

52.75 

11 

51 

55.74 

12 

51 

51.42 

13 

42 

41.77 

14 

39 

35.53 

15 

36 

27.55 

16 

35 

35.46 

LINEAR  R- 

SQUARE 

a  413.5222  ;  PIECEWISE  LINEAR  *  168.978 

CORRELATION  BY 

PEARSON' 

S  METHOD: 

4 

7 

R  » 

0.98 

4» 

507.0000000 

4.0000000 

22.0000000 

*4.0000000 


SPEARMAN  CORRELATIONS: 

R-SUB-S  «  0.96 
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DISTANCE*ERROR  METHOD: 

TAU  =  0.88 


LEXICOGRAPHIC  ORDERING  OP  BENEFITS: 
BENEFIT  NUMBER  OF  ORDER  ERRORS 
1  8 

2  7 

3  7 

4  8 


SOLUTION  1 
VALUE35  80. 4  61 7: 

SUPPORT  ONE  WITH  RESOURCES  AS  FOLLOWS: 
1.366666  OF  CAPITAL 
409.9995  OF  LABOR  1 
0.  OF  LABOR  2 

SUPPORT  TWO  WITH  RESOURCES  AS  FOLLOWS: 
1 .999999  OF  CAPITAL 
40.  OF  LABOR  1 
0.  OF  LABOR  2 


SUPPORT  THREE  WITH  RESOURCES  AS  FOLLOWS: 
14.25  OF  CAPITAL 
0.  OF  LABOR  1 
95.  OF  LABOR  2 

SUPPORT  FOUR  WITH  RESOURCES  AS  FOLLOWS: 
1.  OF  CAPITAL 
0.  OF  LABOR  1  j 

25.  OF  LABOR  2 

DO  NOT  SUPPORT  FIVE. 


SOLUTION  2 
VALUE-80.6269: 

SUPPORT  ONE  WITH  RESOURCES  AS  FOLLOWS: 
1.116665  OF  CAPITAL 
334.9993  OF  LABOR  1 
0.  OF  LABOR  2 

SUPPORT  TWO  WITH  RESOURCES  AS  FOLLOWS: 
1.999999  OF  CAPITAL 
40.  OF  LABOR  1 
0.  OF  LABOR  2 


SUPPORT  THREE  WITH  RESOURCES  AS  FOLLOWS 
9.999999  OF  CAPITAL 
0.  OF  LABOR  1 
66.6667  OF  LABOR  2 

SUPPORT  FOUR  WITH  RESOURCES  AS  FOLLOWS: 
1.083332  OF  CAPITAL 
0.  OF  LABOR  1 
47.0833  OF  LABOR  2 

SUPPORT  FIVE  WITH  RESOURCES  AS  FOLLOWS: 
5.  OF  CAPITAL 
75.  OF  LABOR  1 
0.  OF  LABOR  2 


SOLUTION  3 
VALUE=78. 061 1 : 

SUPPORT  ONE  WITH  RESOURCES  AS  FOLLOWS: 
1.249999  OF  CAPITAL 
374.9995  OF  LABOR  1 
0.  OF  LABOR  2 

DO  NOT  SUPPORT  TWO. 

SUPPORT  THREE  WITH  RESOURCES  AS  FOLLOWS 
12.2045  OF  CAPITAL 
0.  OF  LABOR  1 
81.3636  OF  LABOR  2 

SUPPORT  FOUR  WITH  RESOURCES  AS  FOLLOWS: 
1.545455  OF  CAPITAL 
0.  OF  LABOR  1 
38.6364  OF  LABOR  2 


SUPPORT  FIVE  WITH  RESOURCES  AS  FOLLOWS: 
5.  OF  CAPITAL 
75.  OF  LABOR  1 
0.  OF  LABOR  2 


SOLUTION  4 
VALUE* 77 .3 05 8: 

SUPPORT  ONE  WITH  RESOURCES  AS  FOLLOWS: 
1.499999  OF  CAPITAL 
449.9995  OF  LABOR  1 
0.  OF  LABOR  2 
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LO  NOT  SUPPORT  TWO. 

SUPPORT  THREE  WITH  RESOURCES  AS  FOLLOWS: 
14.25  OF  CAPITAL 
0.  OF  LABOR  1 
95.  OF  LABOR  2 

SUPPORT  FOUR  WITH  RESOURCES  AS  FOLLOWS: 
1.  OF  CAPITAL 
0.  OF  LABOR  1 
25.  OF  LABOR  2 

DO  HOT  SUPPORT  FIVE. 


SOLUTION  5 


VALUE- 71. 9458: 

SUPPORT  ONE  WITH  RESOURCES  AS  FOLLOWS: 
1.116665  OF  CAPITAL 
334.9993  OF  LABOR  1 
0.  OF  LABOR  2 


SUPPORT  TWO  WITH  RESOURCES  AS  FOLLOWS: 
1.999999  OF  CAPITAL 
40.  OF  LABOR  1 
0.  OF  LABOR  2 


DO  HOT  SUPPORT  THREE. 

SUPPORT  FOUR  WITH  RESOURCES  AS  FOLLOWS: 
3.999998  OF  CAPITAL 
0.  OF  LABOR  1 
99.9999  OF  LABOR  2 

SUPPORT  FIVE  WITH  RESOURCES  AS  FOLLOWS: 
5.  OF  CAPITAL 
75.  OF  LABOR  1 
0.  OF  LABOR  2 


DO  YOU  WISH  TO  BEGIN  ANOTHER  RUN  (B)  OR  END  (E) ?  e 
MODEM  TERMINATING  NORMALLY... 


The  five  solutions  provided  by  the  algorithm  have  similar 
subjective  values  but  suggest  very  different  funding  patterns. 
At  this  point  the  decision  maker  would  be  required  to  choose 
between  them  on  the  basis  of  any  non-quanti f iable  information 
or  criteria  he  may  posess.  Sensitivity  runs  with  different 
amounts  of  resources  available  would  point  out  any  important 
discontinuities  in  the  value  of  the  objective  and  facilitate 
the  final  decision. 
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1* Industrie  Canadlenne  de  la 
Montonelge  (1959-1978). 

(November  1978) 


55.  R.A.  More  School  of  Business  Development  of  New  Industrial 

Administration,  Products:  Sensitivity  of  Risk 

University  of  Western  to  Incentives. 

Ontario.  (January  1979) 


56.  Rein  Peterson  Faculty  of  Administrative  A  Study  of  the  Problems  Brought 

Studies,  to  the  Attention  of  the  Business 

York  University.  Student  Consulting  Teams 

Sponsored  by  the  Ontario 
Government's  Small  Business 
Assistance  Programme. 

(February  1979) 


57.  Robert  G.  Cooper  Faculty  of  Management,  Project  Newprod:  What  Makes  a  New 

McGill  University.  Product  a  Winner? 

(July  1980) 

An  Emprical  Study.  Available 
at  $10.00/copy. 

Send  all  orders  payable  to: 

Quebec  Industrial  Innovation 
Centre 

P.0.  Bos  6079 
Station  A 
Montreal,  Quebec 
H3C  3A7 


R.M.  Knight 
J.C.  Lemon 


School  of  Business 
Administration, 
University  of  Western 
Ontario. 


58.  George  F.  Perrie 


Faculty  of  Administrative  Consents  on  the  Course: 

Studies,  Management  of  Creativity  and 

York  University.  Innovation. 

(February  1979) 


59.  J.  Graham  Smith  Faculty  of  Management,  The  Renewable  Energy  Business 

McGill  University.  Sector  in  Canada:  Economic 

Prospects  and  Federal  Government 
Initiatives. 

(May  1979) 

60.  J.W.C.  Tomlinson  Faculty  of  Commerce  &  Cross  Impact  Simulation  of  the 

Business  Administration,  Joint  Venture  Process  in  Mexico. 
University  of  British  (December  1978) 

Columbia. 


61.  Robert  H.  Gras ley  Faculty  of  Administrative  The  Status  of  Innovation  in  the 

Studies,  Strategies  of  Larger  Canadian 

York  University.  Corporations. 

(March  1979) 


Jerry  D.  Dermer  Faculty  of  Management 

Studies, 

University  of  Toronto. 


62.  Z.M*  Rub inski  Department  of  Economics,  The  Small  Firm  in  the  Albertan 

University  of  Calgary.  Oil  and  Gas  Industry. 

(February  1979) 


63.  Don  S.  Scott  Faculty  of  Engineering,  The  Technical  Entrepreneur. 

R.M.  Blair  University  of  Waterloo.  Inventions,  Innovations  & 

Business. 

(1979) 

Available  at  $18. 95/copy 
Send  all  orders  payable  to: 

The  MacMillan  Company  of  Canada 

Limited 

70  Bond  Street 

Toronto,  Ontario 

M5B  1X3 


64.  Harvey  F.  Kolodoy  Faculty  of  Management  Soclotechnical  Study  of  Product- 

Studies,  ivity  and  Social  Organisation  in 

University  of  Toronto.  Mechanical  Harvesting  Operations 

in  the  Canadian  Woodlands. 

(May  1979) 


65.  Richard  T.  Barth 

Faculty  of  Coamerce  and 
Business  Administration, 
University  of  British 
Columbia. 

A  Directory  of  Research  on 
Research. 

(May  1979) 

66.  W.  Ed.  McMullan 

Faculty  of  Management, 
University  of  Calgary. 

Development  of  a  Course  on 
Innovation  and  Entrepreneurship. 
(September  1979) 

67.  Stephen  G.  Pei tc hints 

Department  of  Economics, 
University  of  Calgary. 

Technological  Changes  and  the 
Demand  for  Skilled  Manpower  in 
Canada. 

(January  1980) 

68.  Stephen  G.  Peltchlnls 
Assistance  of: 

Elizabeth  MacDonald 

Department  of  Economics, 
University  of  Calgary 

The  Attitude  of  Trade  Unions 
Towards  Technological  Changes . 
(April  1980) 

69.  Stephen  G.  Peltchlnls 

Department  of  Economics, 
University  of  Calgary. 

Technological  Changes  in 

Banking  and  their  Effects  on 
Employment . 

(January  1977) 

70.  Thomas  E.  Clarke 

Gordon  Laurie 

R.  Peterson 

U.A.  Pleczonka 

T.I.M.E. 

P.0.  Box  6291, 

Station  "J", 

Ottawa,  Ontario. 

K2A  1T4 

Proceedings  of  the  T.I.M.E. 
for  Canada  Workshop. 

(September  29  &  30,  1979) 

71.  Dr.  K.  Paid* 

Dr.  B.  Pazderka 

School  of  Business, 

Queen's  University. 

Background  to  a  Target: 

Approaches  to  a  Valid  Interna t- 
tlonal  Comparison  of  Canada's 
RAD  Expenditures  with 

Particular  Reference  to 

Pha raaceu t icals . 

(April  1980) 

72.  V.H.  Klrpalanl 

N.B.  Macintosh 

Concordia  University. 

Queen's  University. 

Small  Firm  International 
Effectiveness:  An  Exploratory 
Survey. 

(June  1980) 

73.  Dr.  S.K.  Bhattacharyya 
Assistance  oft 

P.H.  Hallett 

Dr.  t.  Bhattacharyya 

An  Assessment  of  Merket  Potential 
for  Intermediate  Capacity  Transit 
System  in  North  America. 

(July  1980) 

74.  O.A.  Ondrack 


♦- 


Faculty  of  Management 
Stud lea, 

University  of  Toronto. 


Innovation  and  Performance  of 
Small  and  Medium  Firms:  A 
Re-analysis  of  Data  on  a  Sample 
of  Nineteen  Small  and  Medium 
Firms  in  the  Machinery  Industry. 
(May  1980) 


75.  T.  Abdel-Malek 


College  of  Commerce, 
Department  of  Management 
and  Marketing, 

University  of  Saskatchewan. 


Canadian  Direct  Investment  in 
Western  Europe. 

(August  1980) 


76.  Stephen  G.  Peltchlnls  Department  of  Economics, 

University  of  Calgary. 


Technological  Changes  and  the 
Sectoral  Distribution  of 
Employment. 

(February  1980) 


Veuillez  falre  parvenir  votre  demande  i  PEIT: 
Please  forward  your  request  for  TISP  reports  to: 

Program  Manager, 

Technological  Innovation  Studies  Program, 
Technology  Branch, 

Department  of  Industry,  Trade  and  Commerce, 

235  Queen  Street, 

Ottawa,  Ontario  CANADA 
K1A  0H5 


J 


